# Good Science is about Measurement ### What are you Measuring? > I like the shift in measurement - how do we measure SGD noise? What is the quantity/thing that we are interested in in the first place. Instead of things like magnitudes of gradients which are a value that could a be proxy of something, you say why not just jump to the thing that we are interested in in the first place. That may take a little bit more time, the actual outcome of the optimization at the end of the day. > > Ask the questions: > * What are we measuring? > * How are we going to measure it? > > A great example would be as follows. Take a network an try training it multiple times. Fix a network (could be a the beginning of training or sometime during training), make a bunch of copies of it and train those copies on different random seeds. Where do these networks then land? Do they land in the same convex region or different convex regions? The way that you figure that out is you take two copies of the network and look at the loss landscape between them. If its flat then they are in the same convex region. If there is a spike then their is a barrier between them. We can think of this as effectively 1 dimensional slices of the loss land scape. The [loss landscape](Loss%20Function.md) is this ultra high dimensional thing and I don't think any attempt to visualize it is very productive. You are just taking this high dimensional thing and trying to compress it down to something much smaller. I don't know if that really works. You are losing a lot of information. What I like here is that you are taking these 1 dimensional slices (which are high fidelity), so you are looking at the loss landscape, you are just looking at these very tiny, narrow slices of a very high dimensional space. But they happen to be very important slices because they relate two different networks to each other. You then need to ask: why is this a good metric? Why does this matter? The answer is that it seems to have descriptive power that allow us to distinguish metrics in other ways. The way that you measure things is still artisanal. If you want to really dig into my papers and rip them apart then look at how I measure things and tell me that is a dumb way of measuring things. --- Date: 20231231 Links to: [Doing Good Science](Doing%20Good%20Science.md) [Measuring the Right Thing](Measuring%20the%20Right%20Thing.md) [Scientific Measurements Use Chains of Proxies](Scientific%20Measurements%20Use%20Chains%20of%20Proxies.md) Tags: References: * [Jonathan Frankle: From Lottery Tickets to LLMs](https://thegradientpub.substack.com/p/jonathan-frankle-lottery-tickets-llms-policy#details) *