Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2947 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.4287. Min Range: -1.4287

Max Act: 1.4287. Min Act: -0.0000

Data Index: 414503 (C4 (Web Text))

Max Activating Token Index: 820

Text #1

Max Range: 1.4287. Min Range: -1.4287

Max Act: 1.4209. Min Act: -0.0000

Data Index: 964817 (C4 (Web Text))

Max Activating Token Index: 473

Text #2

Max Range: 1.4287. Min Range: -1.4287

Max Act: 1.1249. Min Act: -0.0000

Data Index: 1037812 (C4 (Web Text))

Max Activating Token Index: 557

Text #3

Max Range: 1.4287. Min Range: -1.4287

Max Act: 1.0949. Min Act: -0.0000

Data Index: 1011489 (C4 (Web Text))

Max Activating Token Index: 941

Text #4

Max Range: 1.4287. Min Range: -1.4287

Max Act: 1.0990. Min Act: -0.0000

Data Index: 1131218 (C4 (Web Text))

Max Activating Token Index: 166

Text #5

Max Range: 1.4287. Min Range: -1.4287

Max Act: 1.0448. Min Act: -0.0000

Data Index: 92668 (C4 (Web Text))

Max Activating Token Index: 198

Text #6

Max Range: 1.4287. Min Range: -1.4287

Max Act: 1.1589. Min Act: -0.0000

Data Index: 1306540 (C4 (Web Text))

Max Activating Token Index: 755

Text #7

Max Range: 1.4287. Min Range: -1.4287

Max Act: 1.0961. Min Act: -0.0000

Data Index: 1188192 (C4 (Web Text))

Max Activating Token Index: 717

Text #8

Max Range: 1.4287. Min Range: -1.4287

Max Act: 1.0390. Min Act: -0.0000

Data Index: 521973 (C4 (Web Text))

Max Activating Token Index: 817

Text #9

Max Range: 1.4287. Min Range: -1.4287

Max Act: 1.0413. Min Act: -0.0000

Data Index: 1087619 (C4 (Web Text))

Max Activating Token Index: 240

Text #10

Max Range: 1.4287. Min Range: -1.4287

Max Act: 1.0934. Min Act: -0.0000

Data Index: 333458 (C4 (Web Text))

Max Activating Token Index: 756

Text #11

Max Range: 1.4287. Min Range: -1.4287

Max Act: 0.9748. Min Act: -0.0000

Data Index: 510966 (C4 (Web Text))

Max Activating Token Index: 206

Text #12

Max Range: 1.4287. Min Range: -1.4287

Max Act: 0.9764. Min Act: -0.0000

Data Index: 504857 (C4 (Web Text))

Max Activating Token Index: 489

Text #13

Max Range: 1.4287. Min Range: -1.4287

Max Act: 0.9959. Min Act: -0.0000

Data Index: 382802 (C4 (Web Text))

Max Activating Token Index: 253

Text #14

Max Range: 1.4287. Min Range: -1.4287

Max Act: 0.9135. Min Act: -0.0000

Data Index: 988232 (C4 (Web Text))

Max Activating Token Index: 183

Text #15

Max Range: 1.4287. Min Range: -1.4287

Max Act: 0.9582. Min Act: -0.0000

Data Index: 16939 (C4 (Web Text))

Max Activating Token Index: 707

Text #16

Max Range: 1.4287. Min Range: -1.4287

Max Act: 0.9510. Min Act: -0.0000

Data Index: 1353541 (C4 (Web Text))

Max Activating Token Index: 965

Text #17

Max Range: 1.4287. Min Range: -1.4287

Max Act: 0.9251. Min Act: -0.0000

Data Index: 864854 (C4 (Web Text))

Max Activating Token Index: 811

Text #18

Max Range: 1.4287. Min Range: -1.4287

Max Act: 0.9726. Min Act: -0.0000

Data Index: 1270328 (C4 (Web Text))

Max Activating Token Index: 359

Text #19

Max Range: 1.4287. Min Range: -1.4287

Max Act: 0.9651. Min Act: -0.0000

Data Index: 1266461 (C4 (Web Text))

Max Activating Token Index: 371