KNOFLOOK (working title)

KNOFLOOK is an "Auto Factoring" language:
- The base language only allows for concrete code to be created, with no facilities for user defined functions or definitions.
  Creating code is almost entirely done by copy & paste.
- The environment performs a form of parametric compression which results in functions and definitions for redundant pieces
  of code automatically. Any changes in code by the user at the base language level results in a different optimal compression,
  and therefore, a different set of functions. These functions are presented to the user in a way to make the code and its use
  as "readable" as possible.

The base language is a "mostly functional language":
- a set of immutable data types which can only be constructed and accessed:
  * numbers: 1
  * strings "fred"
  * lists: [1, 2, 3]
  * hash tables: ["fred": 1, "john": 2]
  * named tuples: tree(left, right), leaf(val)
- a rich set of builtin operations on these data types
- some APL-like higher order operators for doing simple loops along the lines of map/filter/fold,
  but with friendlier syntax. e.g.
  (filter [1, 2, 3] < 2) == [1]
- a tuple space. This is the language mechanism for dealing with global side effects, concurrency,
  and interfacing with external APIs. More below.
- an optional type system. By default, anything that does not receive a type annotation is dynamically
  typed, and generates only run-time type errors. Since there are no functions, the only thing you
  can annotate are tuple elements. Compile time type errors are only generated if analysis can demonstrate
  100% that a runtime type error will occur given annotations and builtin-operator constraints.

The environment allows editing of KNOFLOOK programs
- a program is in its basic form, a huge expression tree.
- underneath the environment works much like a graphical programming environment, in that the user is
  really manipulating the tree data structure, rather than text which is later parsed (as in traditional
  environments). However, the tree is displayed as something as close to traditional source code as
  possible, to make it as familiar as possible to what programmers are used to.
- the compression scheme does not compress from a fully uncompressed form, as that would be impossible
  to keep in memory at once (imagine taking a large codebase, and textually inlining ALL functions/methods:
  this could result in terrabytes of text easily). Instead, it detects new redundancy using a big expression
  hashtable, and recompresses locally (creates functions), or finds where code is no longer redundant
  (textually inlines functions), see below.
- The environment shows code with a "focus" on a particular subtree. It shows all other code relative to
  this focus, i.e. the environment can filter to show functions used by this subtree first and such.
  This not only helps with code navigation and interpretation, but is also used to determine the scope
  of modifications when they are made to functions used in multiple places (e.g. a modification made to a
  local copy of a function usually results in an additional parameter to the function).
- the internal representation of the code is relatively simple and purely geared towards optimal compression.
  This sometimes results in code that is almost too perfectly factored, hence the environment translates
  between compressed code and programmer readable code.

The following are the rules that govern the compression algorithm (auto factoring):

1. check a tree for duplicates using the hashtable, and if so, make it a shared node. This is just to make the
   identification of possible functions in (2) faster and use less memory
   example: 1+1 becomes a+a where a = 1

2. Create new simple functions:
   a. if a node has a shared child, whose other occurences have the same parent node type, make that node (and the other occurrences)
      into function calls to a shared function with placeholders for all non-corresponding children
      example: a*b+c*b becomes f(a)+f(c) where f(X) = X*b 
   b. if a node has a shared child, and a second occurrence of that child appears inside another child, abstract out that parameter.
      example: a+b*a becomes f(a) where f(X) = X+b*X
   There are few cases where both (a) and (b) apply at the same time, in which case (a) takes precedence.
   
3. if any arg to a function call g is always the function call f, move the call to f inside g.
   This helps with creating bigger functions out of simple ones created by (2).
   example: g(f(1))+g(f(2)) where f(X) = X+1 and g(X) = X*2 becomes g(1)+g(2) where g(X) = (X+1)*2
   
4. unshare nodes whose refc drops to 1
   example: 1+a where a = 2 becomes 1+2

5. inline single use functions, if all args are used once in the body, and inline many use functions that map 1:1 to their body
   example: f(1, 2) where f(X, Y) = X+Y*2 becomes 1+2*2
   example: f(1)+f(2) where f(X) = g(X) becomes g(1)+g(2)

These rules together allow relatively simple compression/refactoring of code without needing complex analysis. 

Combined example:

(3+1)/4+(2+1)/4*3                                     // starting code
(a+c)/b+(2+c)/b*a where a = 3 and b = 4 and c = 1     // rule 1
f(a+c)+f(2+c)*a where f(X) = X/4                      // rule 2a, then rule 4
f(g(a))+f(g(2))*a where f(X) = X/4 and g(X) = X+1     // rule 2a, then rule 4
f(a)+f(2)*a where f(X) = (X+1)/4                      // rule 3
g(3) where g(X) = f(X)+f(2)*X and f(X) = (X+1)/4      // rule 2b, then rule 4


Side effects and evaluation order
---------------------------------

Adding side effects to a language like this is significantly more troublesome than other languages because of the
automated refactoring. The user has no control over where a side effecting expression may end up, and under eager
evaluation this would mean problems both in terms of order of evaluation and number of evaluations.

The only evaluation order that is "neutral" with regards to side effects for refactoring is normal order, hence,
the default order for KNOFLOOK is normal order, with eager evaluation mandated for any purely functional sub-expressions.
Both guarantee depth-first, left to right eval, if you ignore placeholders.

"void" type side effecting expressions (such as print()) are easy: they are naturally intended to be normal order,
they don't make sense in any other evaluation strategy. The big problem lies with side effecting expressions that
also return a value (such as getchar()), where a double occurrence could both be intended as normal order (2 IO ops)
or a reuse of the return value (1 IO op), which would have to be indicated by the programmer. There is no evaluation
order that makes this natural to specify for the programmer, you'd almost need some form of indicating object identity
of the returned value.

KNOFLOOK gets around this by splitting up any such side effects in 2 steps, the actual side effect (the IO op), and
the reading of the result of the IO op. This gives total programmer control over how many times each occurs, without
any special language features.

To allow easy management of all side effecting operations and values, these all reside in a tuple space. A tuple
space is simply an ordered bag of tuples, and expression may insert or read/remove tuples as part of their evaluation
process. We have the following operations:

X => Y   // enqueue X into the tuple space, then evaluate Y as the result of the whole.
X -> Y   // push    X into the tuple space, then evaluate Y as the result of the whole.

push (stack like usage of the tuple space) is useful for dealing with return values of side effects, or complex
tree traversals, queue is useful for ordering of side effects and concurrency tasks.

pop T   // remove the top tuple of type T out of the tuple space (need to think about the syntax... maybe pattern matching is more fun)
top T   // read the top tuple

So, if I wanted to read something once, but use it twice, I could say:

getchar() -> print((top int)+(pop int))

If instead my intent was to read 2 characters and use them each exactly once:

getchar() -> getchar() -> print((pop int)+(pop int))

The good thing is that this code can be refactored in the craziest ways, without semantics being affected.
The environment will generate syntax to remind programmers not used to normal order evaluation what is happening,
by showing calls like f(getchar()) as f({ getchar() }), akin to a block/closure, so it is more intuitive 
that getchar() is not being evaluated at the callsite of f().


What is this all good for?
--------------------------
In short, over the years of programming and seeing countless other programmers at work, I have become convinced
that "perfect factoring" of code is something 99% of programmers are unable to do, or at least don't get the time
for.

Yet, from my perspective, perfect factoring is the #1 programming principle that dominates all others. Almost
any major issues with code can be shown to be factoring problems, predominantly under-abstraction (copy & paste code)
and over-abstraction (bloat, unflexible design).

Refactoring is increasingly hard work the more serious you take it, yet the more perfect you refactor, the more
you are doing the dumb work of a compression algorithm! Even that 1% of programmers could use some help being
more productive by having computers do what they do best, and compression is certainly part of that.

The idea that any programmer of any level of skill would always be working with perfectly factored code, and
would have no choice in the matter, is a powerful one, to me. I can't quite predict the ramifications, it could
range anywhere from "neat toy" to "vast levels of code quality improvement for all mediocre programmers out there".

I have been designing this project for nearly 4 years now, and I think I finally I have it in a shape where it
can work, so I can start on a prototype. Funnily enough the current design is a hell of a lot simpler and yet
more robust than earlier ones, which relied on user editing operations to figure out sharing. Taking the
perspective of pure code compression makes all the difference.



===============================================
===============================================
ADDENDUM:

the above still doesn't work right. If I have:

    getchar() -> print((top int)+(pop int))

And I decide I need to use the outcome of the + twice, i.e.

    getchar() -> print(f(X,X)) where X = (top int)+(pop int)

then this obviously doesn't work. Of course, the environment could signify that X is normal order,
so its the programmers problem, but then the solution becomes to have to write this manually as:

    getchar() -> print(f((top int)+(top int), (top int)+(pop int)))
    
which is terrible.

So really, for maximum refactorability, you want to seperate pop() into drop() and top(), so you get:

    getchar() -> print(f(X,X)) -> drop() where X = (top int)+(top int)
    
This can work, but now you have a push that always needs to be matched to a drop()... might as well
make them into a bracketed structure such that this is done automatically:

    with getchar() do print((first int)+(first int))

this means the result of getchar is only available inside the do. So rather than getchar()
having a return value, it now comes with a block where its value is available!

Can do it even more elegantly: can now actually substitute the getchar() exp instead of first/second!!

    with getchar() do print(getchar()+getchar())

of course, this doesn't cater for 2 seperate getchar()s, but you can have nesting, and escaping:

    with getchar() do with getchar() do print(^getchar()+getchar())

(Actually, with just 1 use theres no need for a with, unless you wanted to change order?)
of course, the variable sub makes all of this more readable:

    with X do print(X+X) where X = getchar()

A combination of with and where should be combined into 1:

    let X = getchar() in print(X+X)
    let X = getchar(), Y = getchar() in print(X+Y)

This also works with "global variables" whose initializers are not unique, in that if I had a myglob = 1, 
then that would simply be a "with" that wraps most of the program. Now, I will probably have more values
of 1 in the program, so if I used this global somewhere deep down, it would essentially be ^^^1 or whatever.
Can even have assignments, i.e.:

    with 1 do with 1 do ^1 = 1+2

    let X = 1, Y = 1 in X = Y+2

The interesting thing is that the composited case now works, if I have:

    with getchar() do getchar()+getchar()

I have now simply created a more complex side effect return value. So if I want to reuse that result, without doing it twice:

    with (with getchar() do getchar()+getchar()) do (with getchar() do getchar()+getchar()) * (with getchar() do getchar()+getchar())

    let X = getchar(), Y = X+X in Y*Y

Alternative syntax? There's no reason to repeat the shared exp in with, alternatively
could just have bracing and an extra escape:

    < ^getchar() + < ^^getchar() * ^getchar() + ^getchar() / getchar() > >

with "^" here means shared at this level, ^^ at the parent, and none means, not shared with anything else.
Though, the syntax doesn't matter much, since the user would never see any of it? you only have
"with" or "^" when you are sharing something, which by definition means it will be autofactored.
    
So how does the user create something a new sharing occurrence? If he copy pastes an exp, and its only
the second one ever in the code base, it will get shared, but will it be a "with"?

Hmm, sounds like there needs to be a way to change back and forth between exps merely shared in code,
as well as shared in result. They will always share code...

Sounds like you want to keep the "with" visible syntax, that way we don't need to get into UI features
to indicate sharing. But you still want this to look something like let-in when code sharing is found,
and it must be different from "where" because it means something different (but only for side-effect-return)...

The "with" syntax is not ideal because it means that already with 1 usage you can create code sharing which
is over-abstraction. Though, the environment could remove this, but only if its manually triggered refactoring.

Also, if you want to represent "with"+"where" as a let-in, you essentially need to introduce a new variable,
to differentiate it from those other uses, yet, its still code shared, you wouldn't want to show the same code
twice under 2 variables! e.g.:

    (X + with X in X+X) where X = getchar()
    
    (X + let Y = X in Y+Y) where X = getchar()

making that a let-in would need Y = getchar() in addition to X = getchar() (duplication), and of course
let Y = X in.. would actually be worse than the above code. But many situations will have the variable
limited to the with scope, so they can have some special syntax, tbd.

What would happen if I had a with myglob in... where myglob = 0 surrounding the whole code, and I typed
another 0 manually somewhere... would it default be converted to myglob? Looks like yes, since it would
search for other occurrences, and then end up giving it the shared name. That would be pretty bad.

Could make it default not shared with myglob, i.e. you'd have to actually type "myglob" to get any
sharing. But since myglob spans all code, it would have to add the new "0" as a with around all code,
with just a single occurrence? actually no, it can make it an inner with. But still, this would mean
potentially tons of single use with's scattered around the code, just to escape the global.
And all of them still would need to refer to a single shared name for the actual value.

let myglob = zero in ... myglob ... with zero in zero ... myglob ... where zero = 0
 
--> Can we solve this problem using... explicit abstraction? Allow the user to name vars explicitly just
to be able to differentiate between different globals? So by default, all zero's would be independent.
Then the user would write "myglob = 0" explicitly (which becomes "myglob is zero"), and then writing
myglob of course would refer to that same 0. This is using identifiers instead of nesting.

Issue is that one good thing about nesting was that it made explicit the creation/deletion time of
the var, which matters because it will cause the var to be reset. So we need an explicit declaration
occurrence that determines the scope.

There really is no problem in allowing users to create variables... as long as the system can still
refactor them fully.

We still need to represent them in some way that is compatible with the idea of an expanded tree.

We could simply have a kind of node that is a "unique writeable location" (indeed unique at this
point since we don't have recursion yet, so they can be mapped to an array of globals!), maybe they'd
look like @0, @1 etc, so you'd have a tree like: let(@1, getchar(), @1+@1). But of course the user
just sees: let myvar = getchar() in myvar+myvar. Also: foreach @2 in vec do..
So, certain trees will have children marked as taking a type UWL, the only difference is that rather
than starting with a value (and later attaching a name), the user types a name and says, make up a
value for me.

Gotta maybe syntactically make a difference between UWLs and constants? That would seem like a good
idea, can just use some syntax coloring.. indicate that UWL's are writable, and that their actual value
is not shown (also gotta make "=" different for both of them, one is an actual assignment op, the other
is a meta-const definition).

Also, we need ways to not have to have "zero" everywhere. And even if it is not a variable with initial
content 0 that needs a seperate name, maybe there's a constant too. So normally a fully explicit subtree
says "this is the only occurrence of this code", and a name says "I refer to 2 copies of this code".
But what if we simply also had explicit code with a particular rendering style that also meant, there's
more than one copy, and functioned identically to a name. The specially rendered explicit is just a
form of a name! by default, any constants would be of such form, but small trees that the user doesn't
want to make up a name for, can be represented such as well. In effect, this is a little bit like my
earliest abstractionless designs, which had whole copies of shared blocks. So in essense, you can now
use both tricks (identifiers and duplicated blocks) to indicate sharing. Though, to keep things less
complicated, any shared blocks with parameters (functions) would be forced to use identifiers? or not
even? A third kind can be the shared folded block, also from the original design, for when you don't
want to give something an identifier, but the block is too big. Infact, by default, any block that
does not have an ident assigned yet can be a shared block, solving the issue of having to come up with
random names too! brilliant! Though, if we going to allow any subtree, then its editing will extend
beyond being just a "name", we have to make sure this falls in line with the new way of doing scoped
editing.. should be no big deal.

--

double check UWLs work with normal order... so:

    let a := getchar() in ...a...a...

this would be clear there is one eval of getchar(), unlike:

    ...a...a... where a = { getchar() }

so {} are shown for side effecting blocks, but the first sample is inline, so not needed.

    X + let a := X in ...a...a... where X = { getchar() }
    
    f(X) = X + let a := X in ...a...a...
    f({ getchar() })

Here, both are shown at once, and it should be clear that getchar() gets done twice?
what about the second case? Yes, since you'd either be looking at f({ getchar() })
with the def below it, or you'd be looking at some sub exp of the def, with X = { getchar() }
below it (when you "zoom in", the env always knows the path from the root, thus the values of
all vars). In both cases you clearly see that X is a normal order var.

If you have a.x := 1, then what if a is not a var (UWL)? This wouldn't work, since it
would expand to say mydatastructure(...).x := 1, which becomes a noop. So language should
forbid it to protect programmers.

For when it is a var, it needs either COW per pointer (since a might point to a data structure
shared with consts), or COW on the first assignment of the whole data structure (probably better).
Can't allow consts to be written into, since that would give unexpected results with 2 pieces of
independent code that both happen to initialize the same shape datastructure.

Copying a var to be used outside its scope of first assignment? This can be allowed, but would
be discovered by the compiler as use before decl.


the environment
===============

use ascii-edit (with refactor on demand) or graph edit (with refactor either on demand or continuous?) ?

continuous seems tricky, as there are situations where something you'd write would instantly be refactored
away again before you can use it twice. Can have a "mild continuous refactoring" which doesn't throw away
such code until a manual hard refactoring, but the constant reshuffling of code while you're creating it
may be disorienting.

Either way, can start with a manual one, and then experiment what it feels like.

Ascii edit CAN work, in the sense that is easy to refresh the entire editing buffer on a refactor. But
there are plenty of issues that don't represent themselves well if the entire thing has to be reparsed
in ascii, with information possibly being lost and hard to reconstruct. It really has to be a tree editor
a la treesheets.


representation
==============

it is easy to represent this code without reference counting, since perfectly factored code doesn't
really have any shared nodes beyond placeholders, and placeholders can be duplicated (referring to a unique ID /
refc placeholders object).

This allows for simple parent & sibling pointers:

enum { T_PLACEHOLDER = 0, T_FUNCTIONCALL, T_INT, T_ADD, T_MUL }

struct Node
{
    Node *parent;           // NULL for the program root and Function::body
    Node *children;         // NULL for T_INT and T_PLACEHOLDER
    Node *sibling;          // in list starting from Node::children, NULL for the last child                   
    Node *nextoccurrence;   // in list starting from Shared::occurrences, only used for T_PLACEHOLDER and T_FUNCTIONCALL
    int type;
    
    union                   // unused for: T_ADD, ..
    {
        PlaceHolder *ph;    // T_PLACEHOLDER
        Function *fn;       // T_FUNCTIONCALL
        int val;            // T_INT
    }
}

struct Shared
{
    Node *occurrences;
    
    bool IsShared() { return occurences->nextoccurrence; }
}

struct PlaceHolder : Shared
{
    PlaceHolder *sibling;   // in list starting from Function::phlist
    Node *currentvalue;     // temp cached value given particular path/focus
}

struct Function : Shared
{
    Node *body;
    PlaceHolder *phlist;
}

struct Selection
{
    Node *n;    // selection is inside of this node
    int start;  // from 0 to numchildren inclusive
    int width;  // idem, - start
    
    int NumNodes() { return width; }
    Node *GetNode(int i) { return width ? n->GetChild(start+i) : NULL; }
}



editing and selection very similar to treesheets, but simpler since it is 1 dimensional.
Copy pasting just clones all nodes, and incs placeholder/functions.
The refactor op (maybe the escape key?) then removes that again.

So rather than deciding between 2a and 2b, we do 2b on everything, then apply 2a on top
of that.



=====================================================================
=====================================================================

Really, we can think of the whole side effect thing even easier:

There's really two languages: the underlying language, and the abstraction language.
We're trying to automatically refactor, not necessarily avoid variables.
The underlying language can pretty directly be imperative.

assume lowercase is the underlying vars, and uppercase is abstraction placeholders:

    { x := 1; print f(x++)+g(x++) }

becomes:

    
    h(X) = f(X')+g(X')
    { x := 1; print h({ x++ }) }
    
    [addendum: x++ will instead be a local function in restructor2, which is better. it has x as a free var.]

we can even graphically indicate placeholders that refer to a normal order arg, here a '

The only special part is that variables have "dynamic scoping" behavior w.r.t. abstractions,
since in the fully unabstracted tree, they effectively have that too (nested scopes in a flat
tree becomes dynamic in an abstracted tree!)
That is not a problem. They don't even need to be shown by the environment, since in the definition
of h(), x is irrelevant, and in the call of h(), the fact that x is used is already clear.

The syntax of the abstraction language should not overlap with the base language at all, in ascii.
For example, a placeholder can be prefixed with a $. In graphics, they can just be marked by
different rendering.

This all also solved the loop problem, i.e. we can now have while/for/foreach/filter/fold/rec/..
with normal variables. It can make the language VERY C like this way. Infact, foreach could almost
just emerge from while:

    { l := [....]; while(l!=null) { foo(head(l)); l = tail(l); } }

if you had that with different starting lists, and different functions for foo, you'd get:

    foreach(L, F) = { l := L; while(l!=null) { F; l = tail(l); } }
    stuff(H) =
        foreach([1..10], { foo(H) })
        foreach([2..11], { bar(H) })
    stuff({ head(l) })

So variables can be pulled pretty far out of context.
    
[in r2:]

    foreach(L, F) = { l := L; while(l!=null) { F; l = tail(l); } }
        helper(F) = F(head(l))
    stuff() =
        foreach([1..10], foo)
        foreach([2..11], bar)

[wow, vastly superior, and no vars out of context!]

For this it seems that having some builtin loops wouldn't go amiss. It might be cool to syntactically show
f(x, {y}) as f(x) {y}, but there will be plenty of cases where f() wasn't meant as a control structure, so
maybe not such a good idea.

Semantically cleaner than the anonymous cell idea.

A mutable data structure simply needs to be regarded as a side effect to get correct semantics,
it returns a seperate piece of memory each time it is evaluated. We can both have mutable and unmutable
data structures, such as strings.
Can integrate heavily with .Net: all .net objects are obviously mutable. Can actually use their vector
classes etc, and all their data types in general. Structs and strings are non side effecting.

Would want to promote a side effect-less programming style, because otherwise the amount of {} will
become quite crazy, and the whole quite unreadable with all these out of context variables?
It will almost be like an automated macro system. Default inline for unnamed functions will help.
Should still try to make things as functional as possible. Even if stuff affected by a var will
be side effecting, the whole block can often be considered functional (if it doesn't contain any
OS calls).

--

Does it make sense to make this an editor for C# code? Probably not.
- currently does not work with classes. restructor code would all sit in a single class, and
  can't handle dealing with existing classes.
  Luckily not a lot of api code appears to require making subclasses, but those that do may
  need to be wrapped in C# first, unless we make a feature for this.
- seeing their lambda syntax allover would be a bit cluttered
- infact, all placeholders would have to become arguments that are now simply omitted
- UNLESS you can take in C# code, refactor it, and output it, and it is still usable in a non-restructor
  environment, it is a bit pointless. Easier to just make a C# code generator, so people can take
  their restructor code elsewhere.

identifier lifting:
at the end of a refactoring op, for each new ident, can look if there's one or more idents that have
lost their function and are now attached merely to a tree, and lift them up (+ concatenate them if there
seperate ones in subtrees). prefix them in some way to make clear its a generated ident.

with such a system, it be possible to do it in ascii first, but still too clumsy?


===============

would be good to allow a system of custom styles for rendering, code preferences etc,
so that people can make it look like whatever language they're used to (will help for
adoption!), not just colors/fonts, but also things like wether to show function calls
in C style, lisp style, pipeline style etc. what kind of brackets to use...

Should then also have an easy option to toggle between default view and custom view,
and make the UI to encourage people to make screenshots etc meant for other people in
the default view.

==================

we can merge our different ideas on side effects, such that we can find out by
experience which is the best.

The base language tries to do as much with pure functions, just because it will
definitely be easiest.

You can then use side effecting variables, as per above, but you'll have to put
up with {} and dynamically scoped use of them everywhere.

As an alternative, there will be FRP blocks. Rather than an all or nothing, FRP
can simply be a feature inside an imperative language: a block of code that
allows read only access to a set of variables from outside it, and applies
effects at the end. This way you can decide yourself how much or how little
goes inside this block, and wrap your frame-loop around it yourself.

You would do this because:
- code would be clearer without {} parameters everywhere
- you want to avoid bugs by seperating data traversal and update
- you want to make your code easy to parallelize

Remains the question of wether regular side effects would be allowed inside such
a block. Thing is, even inside the list of effects there are modifications that
can be invalidated by a previous modification, so the application of effects has
to check for that anyway, so there is no implementation reason not to allow
side effects. But of course if most code is meant to be pure then it may be
nice to have a way where the system can help you avoid side effects, or maybe
just show them graphically even more clearly.

If it turns out that plumbing is very easy, then a 3rd way of doing side effects
is possible, by passing the object around the old fashioned way.

The problem is that now we may need up to 3 ways of writing the same side effecting
operation:

- write(f, "hello")         // side effect
- effect write(f, "hello")  // apply later
- newf = write(f, "hello")  // pure

Though f in the last one would be a placeholder, and a variable in the first two.
So it really depends on the semantics of openfile("input.txt") ?
Seems like the first two might be easier to merge.



=====================================

shockingly, as one of the conclusions out of dynamically_scoped_multimethods.txt,
rules 2a/2b/3 are really all the same: take 2 nodes together into 1 that occur more
than once. This also subsumes 1, so the only additional action remaining is 4/5,
inlining of vars and functions.

Hmm, this does not cover single nodes (like ints), so either those would be taken
as an arg-less function, or shared beforehand, in either case they wouldn't become
args like before... not sure if that's bad.

from a tree pov, these things still look like mini-functions... lambdas.
maybe there's also no reason to have these be multi arg? 

5*3+6 && 7*3+8
f(5)+6 && f(7)+8 where f(x) = x*3
f(5,6) && f(7,8) where g(x,y) = f(x)+y and f(x) = x*3

yes, because taking 2 nodes together sometimes cuts off more than 1 child..
but this is actually kind of cool since it automatically forces any new abstraction
to take over the args of the function it is covering, so no special cases for
argument merging either!

This we call RESTRUCTOR2 !!!

============================================

In FRP_dataflow.txt, we figured we could guarantee the correct amount and order
of side effects by making every side effecting function one that takes an extra
arg to return. This makes a shared side effect into a function rather than an
argument, which forces normal order... can this be applied more generally?
so:

    write("A"); write("A"); 1;

has the danger that the write only happens once if its not marked as normal order {},
but

    write("A", write("A", 1));

does not have that problem as it can only be made into:

    F(X) = write("A", X)
    F(F(1))

which doesn't need a normal order annotation! And you can still use ";" as the
syntax for it.

It fails in particular cases (where the retval is the same, which should be rare
in real code?), and also doesn't work well
with side effects that want to return values like getchar() which are a bigger issue.

but maybe even more solidly can simply always force a side effect to function,
even if it has no arguments at all? 

The weakness of this approach is that if the repeating code is around the side
effect, it requires more names, e.g.:

getchar()*2 + getint()*2

F(X) = X*2
F({ getchar() }) + F({ getint() })

F(X) = X()*2        // would need to show a function is being applied
G() = getchar()     // its a function, but only used once
H() = getint()
F(G) + F(H)         // also hard to see it is a function

or:

F(X) = X*2  
G() = getchar()    
H() = getint()
F({ G() }) + F({ H() })        

however, can use function syntax for:

X = { ... }     // where X is a var from around the focus
X() = ...       // more readable to most people

Still, thinking of X as a function makes using the value X look weird..
though maybe we need a different syntax for a normal order eval X anyway?
Maybe automatically writing it as X() is good even in if we don't adopt
function syntax for leaves.

So yeah, marking args as normal order and making them functions is largely
the same idea anyway.

{ ... }

is short for

lambda() = ...

So that also explains why it is important that users can't edit {} much like they
can't edit functions. If they could, they could remove them, and create a semantics
that requires a variable to inline.

F(X) = X+X
F(getchar())    // user removed {}

let x := getchar() in x*x       // inlined, or when shown as context of focus

that is not too bad? So now we can go both ways on eager vs normal order!
Now we need to track seperately if an exp is side effecting, and make {} language
level syntax, so we can see when to introduce a var.

Also, can make vars and ph's appear to be the same thing, except PHs are purely
functional, so cannot be assigned to.

(note, if we do the X() syntax, would also have to remove the () besides the {})!
Actually, the normal -> eager conversion has to happen in the function, since there
may be multiple callers! and since there maybe be multiple PH uses too, it is best
done on the PH def.

Hmm... so what does this refactor to?

getchar()+getchar && 1+1

does it make sense to present the use with just one function for + ?
certainly in implementation will have to output one copy for each combination
or eager and normal order args, to be efficient.
This situation won't happen a lot in real code, but still needs to be decided
upon.

if we make it one function, then we can't mark normal order in the function
def (or at most: eager vs normal vs "sometimes normal"). It will also allow
the user to remove normal order on a per caller basis. This may be good,
but also bad if theres a lot of callers and you really intended to change
what the function does internally. Definitely need functionality to show
all callers.

If we make it seperate functions (potentially confusing if you think you've
change some pattern, and there's another somewhere).
So the PH def is marked normal order somehow, and the user can mark it eager.

Hmm.. definitely want just one function. No reason why we can't allow
switching the PH (affecting ALL callers), and also one particular caller.

We can still make the syntax X for eager, X() for normal, and X? for mixed,
which will be rare. You can change to eager/normal on any occurrence.

Also interesting for passing on, i.e. { X() } is maybe clearer than { X },
though if we use () can pass it on as just X too?

{} is now a language level node, and can be removed. Though maybe easier
to keep it a special flag? There will already be UI ops to remove a parent,
and to make something eager/normal, so which one is used is kinda...
The flag allows more control, i.e. if there's builtins that take a normal
order arg, it should not be removable. Also can give warnings if you
create a mixed PH this way.

so now as contexts we have:

X = { getchar() }
x := getchar()

to what extend can we start regarding vars as PHs? Could we for example allow
the user to start assigning to PHs?
is the dynamic scope the same?
can we use a let-in also for pure values? or rather, can we represent an assign+vardef
as the usual function call, and just show it as a let if it has 1 caller as usual?

x := 0; while(x<10) x++

F(X) = while(X<10) X++
F(0)

thing is, now X<10 is a side effecting read. So it it got abstracted out, you'd get:

G(Y) = while(Y()) X++
F(X) = G({ X<10 })
F(0)

but that already had to happen with vars. Also, can't be fully inlined, but that is no
big deal. Basically, any op like assignment can't work on constants :)

scope appears like it could be the same too. The old restructor assumed that sibling
subtrees can have same variable names, which does not support true dynamic scope where
2 different variables are used by the same function under the same name. You could
expect to be able to make such a function by copying a call to a function f() that uses
X into another scope that has a different X defined. You could either allow that if
it doesn't break any refactoring algos, or any paste of a tree with vars outside of
its scope needs to make a copy of that value and then refactor (or better yet, move
the defining function up, no can't do that, would break side effects). Refactoring
would create a seperate copy of the var. So if the code being copied is that of the
body of G(), we get new copies of X & Y, and any code that produces it? hmm this sounds
tricky, free vars may rely on other free vars higher up, this could be quite a cascade.
Thing is, true dynamic scope may not be the solution either, since the new destination
may not even have these variables.. so at the very least, even if true dynamic scope
can be supported in some cases, the base behavior needs to be that of either forbidding
the paste op alltogether (not acceptable for code reuse!), or better, allowing it, but
marking the variables as in error (forcing you to copy paste their defs manually, or
other fixes), or do this "deep copy". The deep copy could generate a LOT of code, thus
a lot of work for the restructor to figure out what of it can be shared again. This
may not be desirable. So free vars marked as errors may be a good default, with options
to deep copy, adopt true dynamically scoped var, or paste new def. Having quick fix options
on errors may be a cool idea anyway.


------------------

If we represent vars just like usual PHs as per above, then a variable becomes bound
by an un-inlinable function. So effectively we've allowed the user to sorta kinda
make a function.

If we are going that route, what about just doing full on lambdas? They can be useful
for other things than vars like arbitrary higher order functions.

    5 |> x => x*x

    [1..10] |> map x => x*x

    10 |> for x => print x+y

so => is a lambda like in C#, for/map are lambda operators that turn them into other functions,
and |> is a pipeline / function composition op (from F#, can also support <|)

Very neat and very universal. Bit clumsy looking for straight variable defs, but hopefully their
use will be reduced with this.

With this syntax, could even make for/map normal functions: () around the lambda wouldn't hurt,
and make it look uniform with any other function. Only issue is special functionality like
breaking out of a loop prematurely. Can maybe just implement every pattern, i.e.

exists:    (T -> bool) -> ([T] -> bool)
all:       (T -> bool) -> ([T] -> bool)
find:      (T -> bool) -> ([T] -> object)   // + findlast
findindex: (T -> bool) -> ([T] -> int)      // + findlastindex
filter:    (T -> bool) -> ([T] -> [T])
for:       (() -> ())  -> (int -> ())
makelist:  (() -> ())  -> (int -> [T])
foreach:   (T -> ())   -> ([T] -> ())
map:       (T -> T)    -> ([T] -> [T])
foldr:     (T,T -> T)  -> ([T] -> T)        // + foldl

    5 |> x => x*x

    [1..10] |> map( x => x*x )

    10 |> for( x => print x+y )

Yeah, that looks even better.

Can maybe additionally allow lambda args to be pre-bound, so to make for easier
assignments. effectively like currying:

    (x = 5, y = 6) => x*y

Hmm.. not sure how useful that will be... also, with 2 forms of syntax, have
to take decisions on which is preferable to refactor into...

Also, can make LHS a proper pattern.. the above is a tuple, but with:

    n |> 1 => "one" and
         2 => "two" and
         _ => "something else"

So lambda's can function as a switch-case too!
Maybe the default value is the incoming value, so, you can actualy write stuff like

    _<0 => 0 and _>1 => 1

to write a saturate function!
or:

    openfile("..") |> f!=null => processfile(f)                      // ignore error
    
    openfile("..") |> ( null => return error() ) |> processfile      // not needing a temp identifier to check for null

    openfile("..") | ( null => return error() ) | processfile        // does it look less cluttered this way? a little, but also less active... will make icons for these ops anyway

Special cases:

    n==1 |> true => "equal" and _ => "not"

    n==1 |> true => "equal" else "not"

    n==1 |> then "equal" else "not"

    n==1 then "equal" else "not"
    
    if n==1 then "equal" else "not"

Hmm... Just having special syntax for the latter sounds better. Though an "otherwise" for the last case would be good.

This should allow for better refactoring, as the syntax is highly modular. But will it discover lambdas and higher order
functions?

e.g, two occurences of:

    { [1..10] |> l => while(l!=null) { head(l) |> x => print(x); l = tail(l); } }

    f(X, F) = { X |> l => while(l!=null) { head(l) |> F; l = tail(l); } }
    f([1..10], x => print(x))
    f([2..11], x => save(x))
    
Though likely the original was first defined as:

    { [1..10] |> l => while(l!=null) { print(head(l)); l = tail(l); } }

if you make a copy of this, you'll get a completely new placeholder for l, so not much sharing at all?
First, if you have x => x, can the LHS become an arg? Nope, wouldn't be in scope of anything. So only the
RHS can be restructored. So at best, you get:

    [1..10] |> l => F(l, print(H()))    
    [2..11] |> l => F(l, save(H()))   
            F(L, A) = while(L!=null) { A; L = tail(L); }
            H()[L] = head(L)

So if restructor looks at twonodes |> and =>, he's blocked? no, a => node either only has the RHS as child, or
maybe even more logical, no children at all, and something similar to a Function as type. Lets say Lambda.
So in the above, the => node is similar to a different int. So you cannot share the |>

So how do we go from the above to the more ideal form? Two things have to happen:
- the two l's being equal
- print/save becoming a lambda

That sounds like some kinda lambda-merging op...
if we'd allow the LHS of => also to be taken by a ph, then F(L..) would fully capture l, and then we could allow
a fully captured var to be put inside the body for all callers? Bit of a stretch. Also H() depends on it.

I don't see an easy way in which print would be turned into x => print(x) easily in the current refactoring scheme...

Also shown here: free variables are not necessarily resolved in parents?

interesting side issue: L-value parameters? Might want to mark them in syntax. Will have to compile them as byref anyway.

So, are we only going to allow assignment to lambda args? Otherwise we'll end up with 2 ways to define a function
scope. I guess we have to.

--

Representing lambdas directly is a bit clumsy, but in the end, its just maybe syntactic sugar for functions, just like
the X = 1; ..X.. syntax?

    [..] |> map(x => x*x)

    [..] |> map(F)
    F(X)[A] = X*A
    
this just means one new feature: function as a value as opposed to a call. Unlike the call, we don't have a way to
a way to make the freevars into args, so we actually have to implement them as C# delegates.
Any function value that's used only once can trivially be shown in syntax as => of course.
Since there's no direct caller, they cannot be inlined.
Note: above, inside map, "F" is a function value literal, as different from any vars that happen to have a function
value type. If the code refactors such that a literal follows |>, it can be converted to a normal call, which can then be inlined.

The above example now becomes:

    F([1..10])
    F(l) = while(l!=null) { print(head(l)); l = tail(l); }
    G([2..11])
    G(l) = while(l!=null) { save(head(l)); l = tail(l); }
        
refactored:

    F([1..10])                      
    F(l) = H(l, { print(head(l) })          // can't merge with other head(l), different var
    G([2..11])
    G(l) = H(l, { save(head(l) })
    H(L,X) = while(L!=null) { X; L = tail(L); }
    
L in H would be marked as side effect, simply because it is assigned to.
[refactoring continued below after discussion on R3]

    Remember, the flattest version is:
    
        while([1..10]!=null) { save(head([1..10])); [1..10] = tail([1..10]); }
    
    So the semantics of assignment are: put the value in a cell, so it can be
    overwritten. The only way we know which values share a cell is because they
    are tied by the variable name. So if a variable is SE, any caller PHs it
    directly maps to are also SE, because they would have mapped to the same
    value in the flat version. Any direct SE -> SE ph arg needs to be implemented
    byref. {} becomes a lambda with a byref free var.
    
    Even more basic, while we're at it: 
    
        while(cell(#,[1..10])!=null) { save(head(cell(#,[1..10]))); cell(#,[1..10]) = tail(cell(#,[1..10])); }
    
    So cell(#,[1..10]) returns a reference to a cell, which on first ever access is initialized with [1..10].
    You could explicitly assign #, by number, say, but basically the system does that for you by means of
    the variable the cell was created for. byref allows us to utilize the first placeholder in the chain as
    the cell storage. Or semantically, the moment we have a var def, we turn that into a single caller function,
    and wrap cell(#,X) around the arg, with a new #. If we already have multiple callers the moment we start
    assigning to a PH, all callers get their own #.
    
    The total lifespan of each cell spans a subtree, which is perfectly emulated by the stack calling convention
    of normal functions. In theory though, since there is an exact known number of #'s, we can allocate all
    side effecting vars statically! The amount of memory used is more than a stack, and proportional to the
    size of the expanded code so potentially problematic, but this only for side effecting vars. So more memory
    usage vs lots of byref parameter passing. Also, any {} blocks that only rely on SE vars now don't need to
    be closures anymore.
    
    Actually no: we can do it stack-wise, but allocate the stack statically, with or without normal PH!
    This means free vars are super cheap, and no byref needed at all.
    
    Only big question at this point: Do we ever want to support recursion?
    
    Actually, even if all vars were statically allocated, you could implement recursion by before any recursive
    call, simply backing up the vars into a recursion stack, and going about things as normal. This puts the
    inefficiency on recursive functions only.
    
    - can you statically find out which calls are recursive? yes, you can (a reachable thru algorithm).
      You put all functions and calls in DAG. Any call that breaks the dag gets marked as recursive.
      (not just the breaking calls, also all calls between the return point and the break).
      So a parser would have ALL its functions marked this way.
    - free variables that change from one rec call to the next rather than being global? Worst case you'd
      have to include those free vars in the set to be backed up on the stack.
      A call to a function with free vars is a sub-function, and tends to be non-recursive. a call from
      that function back to the base would not include those free vars. I doubt its an issue.
    
    tailcalls become even more worthwhile now.
    
    So yes, we should implement all vars a static. Quick test in C# shows a static var version to be 20%
    faster than stack based, and that's without taking any free or ref vars into account.
    Code size is a lot bigger, with all metadata references.
    Wrong: if I don't create locals at all in stack based, its 20% faster
    
    So there's 3 variable sizes:
    - all variables of the expanded code, huge.
    - all variables of the factored code, trivial to assign storage, and using memory that's probably more than 10x less than total code.
    - all variables of the factored code that are active at one time (overlapping). Like a static DAG stack allocation.
      Not a trivial algorithm, maybe for later.
      
    One issue about static is that it is maybe bad for garbage collection! you don't clean up vars till much later. The problem is
    present in all 3 of the above, though less in the last, but still..
    Would have to generate code to clear pointer vars programmatically on function exit?
    Clearing all vars makes this style of code 50% slower than stack based. But not all vars need to be cleared, so maybe more like 30-40%.
    Free vars can rectify this, I still think static code can beat closure allocating code by a lot...
    Also, can we statically determine when a pointer var does not need to be reset? Not easily, as that depends on how "soon" it will be
    reused again at a sibling call graph, and its hard to define "soon".
    
    Can do more optimisation: in non-side effecting code, if a caller ph maps directly to a callee ph, they can generally be mapped to the
    same space.
    
    Actually there's a problem:
    
        adder(int x) { return lambda(int y) { return x + y; } }
        add1 = adder(1);
        add2 = adder(2);
    
    here both would refer to the same X. Now the vast majority of closures do not escape their scopes, and this adder example is
    academic in languages that have proper objects, so you could actually specify this semantics as expected. It just makes
    function modifiers impossible, and actually doesn't work with the style of higher order functions suggested above,
    i.e. map(x => x*x).
    
    Also, GUI callbacks is a typical example of escaping scope, and very useful, if you want it to support things like
    making a gui element refer to a particular part of a data structure. It's not needed for traditional "assign action
    to button stuff", and for treeviews, it could do without a freevar if there was a way to query the current element
    the old fashioned way. So to make that work, storing closures in data structures is only allowed if they don't have
    free variables. That is very restrictive.. would have to allow free variables from deeper levels... hard to ensure
    statically, so would have to be done dynamically (resetting inner freevars to null upon escape).
    
    So good idea for a language that needs to be super optimal, compiled to a non-gc environment, or is meant to be a simple language.
    
    Actually can simply restrict function values to args of user funcs and the apply operator, that way no escape analysis
    is needed. Would have to make map() et al actually take a list argument.
    
    C# implements closures by creating a shadow-stack frame object for each function that has its variables used as free vars.
    All variable accesses to a free var, even those in the function of origin, go to this object.
    If you have nested delegates, you can even end up accessing variables thru several indirections of such objects.
    It could instead choose to coalesc two such objects into one, but then it would have the same lifetime problems as above,
    to set things to null explicitly.
    So infact, there's a whole span of options from 1 object per function to 1 object for all functions.
    So maybe you could have 1 object for all functions, with only recursive functions having their own object?
    That would probably be most optimal, yes.
    

===========

R3:

actually, from the discussion above, and the difference between an application operator and a normal function call,
what if instead programs were represented with only apply nodes?
You'd have the apply node, with as first arg something of a function type (builtin or user), and then a whole bunch
of literal nodes, like function literals, and of course still PH's.

The twonode refactor now can't work on a node and a subnode, since they'd both be apply nodes, likely.
Instead, the same effect would be achieved by abstracting any apply nodes that have two children in common.
(tempting to do "two or more", but that might actually inhibit the optimal refactoring, given frequency of two-combinations).

In the original scheme you actually take together 2 node types, not nodes. In this one, you can limit yourself to everything
except apply nodes, though there's no reason other than efficiency: if the most frequent twonode is one where one of the
two nodes is a deep tree, then clearly that is a good refactor.

does it give the same results?

    5*3+2 == 4*3+2

in R2:

    F(5) == F(4)
    F(X) = X*3+2
    
here:

    (== (+ (* 5 3) 2) (+ (* 4 3) 2))

    F(X) = + X 2
    G(X) = * X 3
    (== (F (G 5)) (F (G 4)))
    
in R2, we'd now take F and G as a twonode... but here that doesn't work?
We'd have to revert to R1's rule #3 again?
That may not be too bad, at least rule 2 has no a/b, so not ambiguous.
So in terms of simplicity, R2 > R3 > R1.

Also have to enter into the hashtable every permutation of two args, or
fib(nargs). That gets expensive for higher args [0, 1, 3, 6, 10, 15,...]
Though that's the price you pay for most optimal refactors. If you you have
a unique call in your code with 6 args, that will clutter the hashtable with
15 entries on every refactor. Luckily the vast majority of functions is 1-3
args.

    Hmm could convert the whole thing into forth format, and then 2 & 3 would be
    the same again?
    
        5 3 * 2 + 4 3 * 2 + ==
    
    Now twonode means take any consecutive two:
    
        5 F 4 F ==
        F = 3 * 2 +
    
    But forth can't deal with holes in the middle:
    
        3 2 * 3 4 * +
    
    That has no two consecutive reps, but it clearly has a rep.
    You can't just recognize things with a 1 gap difference, as 2 and 4
    can be arbitrarily complex, and different length.
    You'd have to enter every permutation again into the db, which
    becomes doable for a highly factored program, but even in a factored
    program, there may be long strings of unique stuff (such as
    data initialization), and n! would kill that.
    So that can't work. I don't see any other way to cheaply find (3 X *).
    Witness... the power of trees :)
    Also, you'd still have to deal with placeholders, and they'd be different
    from stack arguments. So not a lot of simplicity gain.
    Stack based is therefore a lost cause. qed.
    Actually, an interesting idea, especially if you just think binary op
    centric for the moment, is that every function literal has an alterantive
    version that is the same but its args swapped (for non-assoc functions
    anyway). So the program could do a two-conseq refactor, swap args, and
    try again. That potentially breaks optimal refactor in terms of freq,
    might give problems with evaluation order if there are side effects,
    and would mean you'd have to build a language entirely out of binary ops,
    but it could work.
    
        2 F 4 F +
        F = 3 *'
        
    Behold, programming reduced to string compression :)
    More shockingly, this stack language doesn't need any swap ops, as swaps
    are encoded in the operators.
    Though, even if you only did binary ops, any user F could take arbitrary
    number of args, so those could not be part of the swapping scheme which
    would be pretty limiting. Also, the body of F could have operators whose
    args are outside of F, so those too can't be swapped if F can't be swapped.
    Limiting the body of F to 2 elements won't work, since you can still have
    + + as the body.
    And saying +' is really the same as (swap +). You could do this on a regular
    forth by trying to insert swap in front of every op, swapping the args if
    possible, refactoring while taking swap as a required middle element of
    the twonode, then removing swaps again where possible.
    Could then do seperate passes for 3 arg funs. Maybe decide 4 or more
    args simply doesn't get this refactoring.
    Would be a decent auto-refactor system for a forth language, but can't
    compete with tree based. Certainly not a good underlying system for
    a tree based system.
    
        [1..10] F
        F = { dup null != } { dup head print dup tail assign } while
        [2..11] G
        G = { dup null != } { dup head save  dup tail assign } while
            
    so assign uses a stack location as a cell... I guess that can work...
    {} is a quoted single use fun

        [1..10] F
        F = H { I print J } while
        [2..11] G
        G = H { I save  J } while
        H = { dup null != }
        I = dup head
        J = dup tail assign
        
    That's how far it gets without swapping. And then it's stuck, since
    you can't swap on J for example, because I and print are not isolated
    subtrees.
        
    So of theoretical interest only.

So back to a potential R3, what is better about it? 
Can deal with functions as values more uniformly.

    (F [1..10])
    F(l) = (while (!= l null) (seq (print (head l)) (assign l (tail l))))
    (G [2..11])
    G(l) = (while (!= l null) (seq (save  (head l)) (assign l (tail l))))

with purely rule #2 and #4:

    (F [1..10])
    F(l) = (L (K (print J)))
    (G [2..11])
    G(l) = (L (K (save  J)))
    J = (head l)
    K(X) = (seq X (assign l (tail l)))
    L(X) = (while (!= l null) X)

[wrong, this forgets that l's are different, see below]

apply #3:

    (F [1..10])
    F(l) = (L (print J))
    (G [2..11])
    G(l) = (L (save  J))
    J = (head l)
    L(X) = (while (!= l null) (seq X (assign l (tail l))))

Funny that rule #3 was defined on the function head, but here that is no special value
anymore, really the rule is: if a child has a child that's always the same, flatten
the args such the the apply happens inside the function. So the steps would have been:

    (F [1..10])
    F(l) = (L K (print J))
    (G [2..11])
    G(l) = (L K (save  J))
    J = (head l)
    K(X) = (seq X (assign l (tail l)))
    L(X, Y) = (while (!= l null) (X Y))
    
    (F [1..10])
    F(l) = (M (print J))
    (G [2..11])
    G(l) = (M (save  J))
    J = (head l)
    K(X) = (seq X (assign l (tail l)))
    L(X, Y) = (while (!= l null) (X Y))
    M(X) = L K X

    (F [1..10])
    F(l) = (M (print J))
    (G [2..11])
    G(l) = (M (save  J))
    J = (head l)
    K(X) = (seq X (assign l (tail l)))
    M(X) = (while (!= l null) (K X))

    (F [1..10])
    F(l) = (M (print J))
    (G [2..11])
    G(l) = (M (save  J))
    J = (head l)
    M(X) = (while (!= l null) (seq X (assign l (tail l))))

But now we can do the same rule to J:

    (F [1..10])
    F(l) = (M print)
    (G [2..11])
    G(l) = (M save)
    M(X) = (while (!= l null) (seq (X (head l)) (assign l (tail l))))

Now the last step we need is to see "l" not as a free var, but as
an explicit var. If that were the case, we could apply #3 to M and l:

    (F [1..10])
    F(l) = (N print)
    (G [2..11])
    G(l) = (N save)
    M(X,l) = (while (!= l null) (seq (X (head l)) (assign l (tail l))))
    N(X) = M X l

Hmm, that is a refactoring that would loop forever.
Better yet is to say if we have vars that are not used at all in the body,
and the freevar is used by only one function inside of it, move it there:

    (F [1..10])
    F(l) = (M print l)
    (G [2..11])
    G(l) = (M save l)
    M(X, l) = (while (!= l null) (seq (X (head l)) (assign l (tail l))))

Now we need to see that l in M only makes l in F/G SE if it is used more than
once. If not, we can inline:

    (M print [1..10])
    (M save [2..11])
    M(X, l) = (while (!= l null) (seq (X (head l)) (assign l (tail l))))

which is the final desired form. To get here, we needed to use:
- the newly defined rule #3
- not allow a side effect to travel further than it needs to, so inlining can happen.

That is not TOO bad.

But what if the save function had an extra arg:

    (F [1..10])
    F(l) = (M (print J))
    (G [2..11])
    G(l) = (M (save 1 J))
    J = (head l)
    M(X) = (while (!= l null) (seq X (assign l (tail l))))

now rule #3 clearly doesn't work anymore.
Only way to do this, is to define it as:
rule 3 v3: if all callers of a function have a child that all have the same child (J),
make that child into a function, then continue as old rule 3:

    F(l) = (M N [1..10])
    G(l) = (M O [2..11])
    M(X, l) = (while (!= l null) (seq (X (head l)) (assign l (tail l))))
    N(X) = print X
    O(X) = save 1 X

Which begets the same desired result, albeit more generically.

doh, all this while we have not dealt with the l's being different, e.g.:

    (F [1..10])
    F(l) = (while (!= l null) (seq (print (head l)) (assign l (tail l))))
    (G [2..11])
    G(i) = (while (!= i null) (seq (save  (head i)) (assign i (tail i))))
    
refactoring that:

    (F [1..10])
    F(l) = (H (print (head l)) l)
    (G [2..11])
    G(i) = (H (save 1 (head i)) i)
    H(X, L) = (while (!= L null) (seq X (assign L (tail L))))

Now even the above refactorings don't help, since besides H and head, the two exps don't have a lot in common.
You could say that if you're passing a side effecting var, and a normal order exp that relies on that side
effecting var (for all callers), that is a sign that what you are doing really works best with a lambda,
so first step would be to pull the var out of the other exp:

    (F [1..10])
    F(l) = (H (J l) l)
    (G [2..11])
    G(i) = (H (K i) i)
    H(X, L) = (while (!= L null) (seq X (assign L (tail L))))
    J(X) = (print (head X))
    K(X) = (save 1 (head X))
    
now rule #3 can apply:

    (F [1..10])
    F(l) = (H J l)
    (G [2..11])
    G(i) = (H K i)
    H(X, L) = (while (!= L null) (seq (X L) (assign L (tail L))))
    J(X) = (print (head X))
    K(X) = (save 1 (head X))

And get to the ideal situation easily. This required:
- rule #3 v2
- rule #6 new: create lambda from normal order arg, if it relies on a var thats already being passed

- rule #3 (v2) should really be 2 super generic steps:
  3a: move an application into the body if the var is used once, and all callers have same #args (problems with unifying types, potentially)
  3b: move equal args across callers into the body
  Or even simpler: move anything that's the same accross all callers into the body, wether its an application or otherwise, if the var is used once.

So that's all not too bad... but can we apply some of these ideas to R2, and get it simpler?

    F([1..10])                      
    F(l) = H(l, { print(head(l)) }) 
    G([2..11])
    G(i) = H(i, { save(1, head(i)) })
    H(L,X) = while(L!=null) { X; L = tail(L); }

rule #6:

    F([1..10])                      
    F(l) = H(l, { J(l) }) 
    G([2..11])
    G(i) = H(i, { K(X) })
    H(L,X) = while(L!=null) { X; L = tail(L); }
    J(X) = print(head(X))
    K(X) = save(1, head(X))

rule #3a is now not as obvious since it needs an application operator, but extra desirable since the arg is normal order. Directly followed by 3b:

    F([1..10])                      
    F(l) = H(l, J) 
    G([2..11])
    G(i) = H(i, K)
    H(L,F) = while(L!=null) { L |> F; L = tail(L); }
    J(X) = print(head(X))
    K(X) = save(1, head(X))

G/F inlined:

    H([1..10], J) 
    H([2..11], K)
    H(L,F) = while(L!=null) { L |> F; L = tail(L); }
    J(X) = print(head(X))
    K(X) = save(1, head(X))

still haven't inlined head(), this would be a new refactoring in both R2 and R3.
Normally a function has multiple callers, but with applications, it occurs that
a call has multiple functions! So we need to do reverse refactorings on that.
In this case, if the functions all have the same context of a var, it can be
moved into the caller. This would be rule #7.

So in conclusion:
- both models need about the same amount of additional refactorings to make complex
  cases happen
- R3 doesn't show clear advantages over R2, and some disadvantages. Not convincingly
  worth it. 
- We'll be able to use any function as value, have an apply operator, and assign to
  any var. Lambdas are syntax. Function values by themselves allow all kinds of
  loops. We can implement let in terms of lambda, but assignment is an orthogonal
  feature to that, which we want. We can do a lot with let+loops, so we could
  postpone it? Being able to infer new iterators from assignment code requires
  a LOT of new refactorings and assignment analysis, so maybe best for later.


lambdas vs normal order blocks?
one is user delimited, the other is naturally refactored, so maybe shouldn't be displayed
the same, and () => exp is uglier than { exp }, we could consider => exp, but the thing is,
its not really like a lambda since it is not factored into a function and apply. It's not
wise to do that anyway, since the side effect status can change with a single edit, and
we don't want to have to restructure the code when that happens?

{ x | x*x } is cool syntax, and the {} are welcome delimiters, but they don't work well
with higher order functions (surrounded by an addition ()).

Maybe a backquote ` for normal order blocks? too invisible, and too unlike C. {} is good
because its sorta familiar, it stands out, and it should remind people about side effects,
whereas => is not necessarily about side effects (even though both are about non-eager eval).

They both need to be closures with free vars, so we'll use the same tech underneath.




==================


big typesystem issue, if we have
"" + "" + ""
1 + 1 + 1
that will give a function:
F(X, Y, Z) = X + Y + Z
and all ints will have to go thru strings!
we can give concat a different op, but its an issue for float and int too

2.0 + 1
2 + 1
So individual calls want a union between args and body,
but neither a union nor an intersection can work between callers.
basically
F(X) = X+1
can never be typed correctly without templated args

So we can can typecheck as if we are typechecking the expanded tree, by typechecking
the function for each call independently. PHs are simply set to arg types.

We can speed this up by checking if a call has the same arg types as a previous type
check, if so, it shares its results. So functions keep a lists of "variants" that
have sets of args and ret types. Each call then selects a particular variant.

The cool thing is that this simplistic type system is able to statically type
by 100% type inference what other languages need pretty complicated type systems for.
Though function values will need special care: easy with builtin operators, but not
in the most general case.

One issue is type errors, they exist only in certain variants now. Since calling
user functions never gives a type error now, all errors are between builtins and
their arguments, or in the case of a PH, the arg for that specific function call.
So builtins should tag errors to their args, such that if its a PH, it can
perculate up to the arg that creates it.

easiest way to store variants it is to just keep a list of caller nodes, as they
hold all information.

Have to actually compile all variants to seperate bytecode, as the CLR does not
support operators like + etc on generically typed args, nor does it allow int/float
as constraints on them. This could bloat up the code quite a bit, but generally,
compiling a function multiple times should not necessarily require the functions
called therein to be compiled multiple times too. Variants are not going to
happen a lot, the worst case being some large piece of code which someone calls
with both an int and a float arg.

So, WOW, this system:
- allows code with the flexibility of dynamic typing (can call
a function with ANY datatype as long as the builtins can work on it),
- is fully strongly typed
- yet needs no complex TI or generics
- exact error messages pointing to a value and a builtin, no type parameter stuff etc.
- cannot fail at runtime (no dynamic type checks of any kind)
- and is fully optimal (all code works on exact types), can be specialized easily

It is not actually fully dynamically typed, i.e. control flow like F(true ? A : B)
where F only accepts A's in not taken into account, as are change of variable type
once assigned, or heterogenous lists etc. But at least it does bounded generics
fully automatically. It can be extended, i.e. typechecking such if-thens by
generating 2 variants of F would be possible.

What about sub-super types in objects? if I already have a variant on a super type,
can I reuse it? As long as we don't have any builtins that are overloaded on sub & super,
yes. Problem is with code that calls .net api functions with overloads. Maybe we'll
have to mark these variants as such, to disallow variant reuse.
But what if I first encountered the subtype? I am then going to have to re-type check
for the supertype. If that succeeds, I can then remove the subtype variant so both
will compile to the same code. Same caveat on using overloads.

Function values screw up this scheme pretty good, as the type of x in:

    [1...] |> map(x => print x)

can only be known from the list. 
So |> is special from all other nodes, it gets it LHS normally, then sets that type
to its RHS. It should have a special typecheck function for each kind of node for
such a RHS, which is kind of like a reverse typecheck.

        normal()            apply-RHS(T)
        
PH      ret type            match type
BI      match children      pass on T...
FV      ret type            like a normal function call

so the reverse type checking ends at a body of a lambda.

sometimes the apply is internal:

    dostuff(x => print x)
    
So type checking of the arg is delayed, and the type later set by the |>


===========================

what is flat representation of a function value?

you could see

    map(x => x*x)
    
as:

    map($[..] * $[..])

With $ indicating which are to be subtituted by whatever map does, and
wrapped in a cell in the most general case with an assignment involved.
Except... [..] may not be readily available:

    (if .. then $[..] else $[..]) |> map(x => x*x)

is then really 

    if .. then map($[..] * $[..]) else map($[..] * $[..])
    
so the RHS of an |> wraps around the LHS, so maybe it was:

    map($(if .. then [..] else [..]) * $(if .. then [..] else [..]))

That makes more sense. It is then map who says, I am going to repeat
eval this exp and grab a different value from it every time.

That doesn't necessarily explain how we get from exps with $ in them
to a function value, but at least all code can be represented.

And it explains the inverse type checking pretty much.

   
========

alternative ways to represent recursion

we can bind recursion to lambdas, that way we don't have to name them.

    fac = n => if n==1 then 1 else n*REC(n-1)

the => / REC can span several normal function calls, as REC always refers
to the closest lambda, not to regular functions.

Can't quite implement parsing, as that has nested/overlapping recursions...

Though can pass recursive functions to eachother:

parsefactor = pe => ... parsefactor |> pe ... REC() ...
parseexp = () => ... parseexp |> parsefactor ... REC() ...
() |> parseexp

This might give typechecking problems as the two types
now rely on eachother... but that may be fixable.

Also, maybe instead of REC just using its own function value
may work.



============================

side effects analysis:

problem: the system cannot distinguish between F(SE) where SE was originally
inside the body, and where it was written by the programmer already in arg
form.

Also, we wanted the ability for the programmer to change meaning by removing
the normal order.

Also, there are plenty of situations where normal order is not necessary,
i.e. if the SE var doesn't get changed between the moment the arg gets evaluated
and the normal order eval takes place.

So really, adding normal order is something that should happen at refactoring
time, as the only thing we're trying to do is prevent a refactoring changing
meaning. What happens beyond refactoring is entirely up to the programmer.

So maybe we should use the () => syntax, more uniform, easy to remove, and shows
we made something into a function purely to preserve evaluation order.

The use of the PH should be a function call to preserve uniformity... that makes
switching a bit harder however, so maybe a dedicated function is needed.

maybe:

    X       eager eval
    X!      normal order eval
    X?      normal order or eager
    X!(a)   function application

--

The actual refactoring:

Since the twonode refactoring preserves the order of eval of its eager args,
we only need to care about if one of the two nodes is a side effect.

(here we replace = by -> to keep the order of eval post order)

    X -> var        // does not change order of eval
    const -> X      // where X must be a var by ref, so does not change order of eval
    
    X + (Y -> var)    // does not change order of eval
    (X -> var) + Z    // if Z contains an occurrence of var, we have a problem

-> + can be taken together as either:

    (X -> var) + Z
    F(X) + Z where F(X)[var] = X -> var

depending on wether it was abstracted before.
So really, the top level two-node must collect assigned to vars from left to
right, and then see if they are used in subsequent nodes. If so, such a node
must be made into a () =>.
[actually its not left to right, it is the lower of the twonode vs any subsequent
new PH values]

instead of:

    G(X, Z) = (X -> var) + Z
    G(1, var)

we get:

    G(X, Z) = (X -> var) + Z!
    H()[var] = var 
    G(1, `H)

must definitely be displayed inline. H occurs once, but one of its "callers" is
a function value, which cannot be inlined. Only if we get a function value
next to an apply operator, they can be converted to a normal call, which then
can inline the function.

We can use ` for the function value operator? Not sure if its needed if we always
show these functions inline.

--

How does this relate to framework calls? If we assume such a call has a nodetype
that indicates the method, and then a list of args normally (if the method is non-
static, then the first arg be the this variable, etc).

    Console.WriteLine("A"); Console.WriteLine("B")

Can happen as:

    Console.WriteLine(X); Y     // problem
    X; Console.WriteLine(Y)     // ok
    
This is the same configuration as assignment above. The problem is, that we
don't know what variable it is side effecting on. If we want to be fully safe,
we must assume that all framework calls modify a common state. But probably
safe enough is to say static calls modify a common state, or even static calls
on a particular class ("Console"). non-static calls could possible be assumed
to have side effects only on the *contents* of a variable.

Btw, if a framework method takes ref/out parameters, those should of course be
taken as an l-value side effect. Infact, assignment itself is nothing but a
builtin with an out parameter.

So our list of left to right side effects must be able to contain different
things:
- variable l-value (assignment/ref/out -> use)
- variable contents (content use -> content use)
- class statics (use -> use)

We could have an option for even safer refactoring that considers all framework
code dependent, but we should investigate if there are examples of:
- two static classes that have dependencies
- non-static methods that access statics.

If we had even more information that be good, it be a shame if two read-only
ops would cause the second to be normal order. We can either try to annotate
this stuff for commonly used classes, or wrap a lot of functionality such that
direct use of the framework is not often needed... though that sounds impossible
for gui stuff. Another more advanced idea is to write a bytecode analysis tool.
It can be conservative, i.e. by default it assumes full side effects, and 
if either an annotation or a byte code analysis proves its otherwise, then that is
used instead.

So we have read & write side effects. What we're really looking for is either
a WSE in the lower twonode vs a RSE or WSE in the higher twonode remaining args,
or a RSE in the lower vs a WSE in the higher.
And a WSE always dominates any SE on content.

Have to track SE's thru aliases (passed to another var), and content SE's even
thru data structures. Some of this may be too hard, in which case a conservative
estimate is made (any var written to a data structure counts as a side effect).
Same with 2 branches of an if-then.



