outlines the choices, condiderations and tradeoffs made
STATUS: Active development
AUTHOR(S): Jovan Sardinha (my_first_name.my_last_name[at]gmail.com)
CREATED: 2020-03-19                               
UPDATED: 2020-05-25

Background

Objective

New ideas, concepts and tools are constantly being created. Doing literature reviews and one off experiments for every new problem is painful. Can we create tools that can help the learning process but then be used later to solve related problem?

Motivation

How do you figure out what do you spend your time learning?

When learning new concepts and idea, it is easy to glaze over the detials and get a false sense of confidence - but the devil is always in the detials. A lot of new concepts might seem intuitive, however, actually understanding tradeoffs, decision and limits is hard from initial reading.

How to improve the decision making process?

When trying to solve a new problem, a lot of ideas come to mind. However there is a lot of friction in prototyping all these alternative ideas. Hence, a lot of shortcuts are taken leading to sub-optimal outcomes in the long run.

Can one auto-catalyze the learning, decision and implimentation process?

Having a good aresenal of ideas, improves the quality of decisions. Making better decisions helps create better solutions. Creating better solutions should give hints/ideas what to learn next.

Design

Below are the design choices that I made:

  • Using nbdev as a base structure

    • ✅ Will use this to get something off the ground. However, will not tie too deeply into this strcuture (if I need to swap this out).
    • ✅ Nice literate environment
    • ✅ Get PyPI functionality out of the box
    • ✅ Get docs functionality out of the box
    • ✅ Get unit test functionality out of the box
    • ❌ Might need to contribute to nbdev to add functionality
      • 🛡 clone an internal verison of nbdev
      • 🛡 write all non-supported use case directly in the lib folder

  • Monolithical git repo

    • ✅ Will allow for incrementability
    • ✅ Will become canonical place with experimentation/learning will happen. This will help with compounding learning/experimentation speed.
    • ❌ Will enentually get hard to maintain?
      • 🛡 things don't have a lot of deps will eventually fall out/depricate
    • ❌ >0 startup time

  • Lack of structure

    • ✅ Hyrum's Law
      • hard for others to understand/follow
        • 🛡 build tools to show dependency
    • ✅ allow for incrementability

  • Progressive disclosure of complexity

    • ✅ easy to start
    • ✅ allow for rough initial ideas to stable APIs to co-exist
    • ❌ A lot of extra work needed to trace dependencies
      • ️️🛡 build better tools to view dependencies

  • Multi-language/multi-framework/multi-domain support.

    • ✅ This becomes the canonical repo for learning/experimentation.
    • ❌ Only python will be used in notebooks (for now and because of nbdev limitation).
      • 🛡 write all non-supported use case directly in the lib folder.
Legend:
✅ suppotive/positive point
❌ fault/negative point
🛡️ mitagation plan

Alternatives Considered

  • Use a collections of notebooks to store ideas

Performance Implications

  • Single package bloat
    • Inefficient when trying to productionize
    • Cost to maintain loose ends and brokes paths

Dependencies

A list of all package dependencies are avaiable in the settings variable in the settings.ini file.

Open Ended Questions

  • Is there a way to track how effective this tool is for
    • the learning process
      • speed of learning new ideas?
      • complexity of ideas being learnt?
    • the decision making process
      • quality of ideas considered during the decision making process?
    • the implimentation proces
      • time from concept to final product?
      • reliability of solution?

References