Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 646 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.5786. Min Act: -0.0001

Data Index: 1325348 (C4 (Web Text))

Max Activating Token Index: 53

Text #1

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.4536. Min Act: -0.0000

Data Index: 1244751 (C4 (Web Text))

Max Activating Token Index: 31

Text #2

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.3922. Min Act: -0.0000

Data Index: 1011280 (C4 (Web Text))

Max Activating Token Index: 469

Text #3

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.4303. Min Act: -0.0000

Data Index: 275173 (C4 (Web Text))

Max Activating Token Index: 748

Text #4

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.3710. Min Act: -0.0000

Data Index: 88598 (C4 (Web Text))

Max Activating Token Index: 739

Text #5

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.2331. Min Act: -0.0000

Data Index: 274879 (C4 (Web Text))

Max Activating Token Index: 965

Text #6

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.2460. Min Act: -0.0000

Data Index: 966009 (C4 (Web Text))

Max Activating Token Index: 777

Text #7

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.2733. Min Act: 0.0002

Data Index: 1253205 (C4 (Web Text))

Max Activating Token Index: 86

Text #8

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.2140. Min Act: -0.0001

Data Index: 1003682 (C4 (Web Text))

Max Activating Token Index: 466

Text #9

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.1585. Min Act: -0.0000

Data Index: 348618 (C4 (Web Text))

Max Activating Token Index: 183

Text #10

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.1862. Min Act: -0.0000

Data Index: 1107035 (C4 (Web Text))

Max Activating Token Index: 474

Text #11

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.1659. Min Act: -0.0000

Data Index: 469681 (C4 (Web Text))

Max Activating Token Index: 153

Text #12

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.1806. Min Act: -0.0000

Data Index: 1217429 (C4 (Web Text))

Max Activating Token Index: 103

Text #13

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.1576. Min Act: -0.0000

Data Index: 382873 (C4 (Web Text))

Max Activating Token Index: 752

Text #14

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.1045. Min Act: -0.0000

Data Index: 1295811 (C4 (Web Text))

Max Activating Token Index: 79

Text #15

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.0751. Min Act: -0.0000

Data Index: 460090 (C4 (Web Text))

Max Activating Token Index: 994

Text #16

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.0828. Min Act: -0.0000

Data Index: 865065 (C4 (Web Text))

Max Activating Token Index: 231

Text #17

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.0270. Min Act: -0.0000

Data Index: 1078908 (C4 (Web Text))

Max Activating Token Index: 825

Text #18

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.0374. Min Act: -0.0000

Data Index: 256315 (C4 (Web Text))

Max Activating Token Index: 101

Text #19

Max Range: 1.5786. Min Range: -1.5786

Max Act: 1.0641. Min Act: -0.0000

Data Index: 698865 (C4 (Web Text))

Max Activating Token Index: 492