Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 325 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.2375. Min Act: -0.0000

Data Index: 685274 (C4 (Web Text))

Max Activating Token Index: 209

Text #1

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.2399. Min Act: -0.0000

Data Index: 1201606 (C4 (Web Text))

Max Activating Token Index: 304

Text #2

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.2659. Min Act: -0.0000

Data Index: 307392 (C4 (Web Text))

Max Activating Token Index: 64

Text #3

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.2573. Min Act: -0.0000

Data Index: 406774 (C4 (Web Text))

Max Activating Token Index: 394

Text #4

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.2292. Min Act: -0.0000

Data Index: 1020345 (C4 (Web Text))

Max Activating Token Index: 60

Text #5

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.2305. Min Act: -0.0000

Data Index: 1088012 (C4 (Web Text))

Max Activating Token Index: 492

Text #6

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.1639. Min Act: -0.0000

Data Index: 798939 (C4 (Web Text))

Max Activating Token Index: 681

Text #7

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.2517. Min Act: -0.0000

Data Index: 163317 (C4 (Web Text))

Max Activating Token Index: 257

Text #8

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.2321. Min Act: -0.0000

Data Index: 1189050 (C4 (Web Text))

Max Activating Token Index: 977

Text #9

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.2045. Min Act: -0.0000

Data Index: 1135591 (C4 (Web Text))

Max Activating Token Index: 746

Text #10

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.2087. Min Act: -0.0000

Data Index: 1138660 (C4 (Web Text))

Max Activating Token Index: 973

Text #11

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.1745. Min Act: -0.0000

Data Index: 1127333 (C4 (Web Text))

Max Activating Token Index: 511

Text #12

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.2024. Min Act: -0.0000

Data Index: 762521 (C4 (Web Text))

Max Activating Token Index: 169

Text #13

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.2546. Min Act: -0.0000

Data Index: 360035 (C4 (Web Text))

Max Activating Token Index: 747

Text #14

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.1880. Min Act: -0.0000

Data Index: 1335266 (C4 (Web Text))

Max Activating Token Index: 287

Text #15

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.1063. Min Act: -0.0000

Data Index: 1085924 (C4 (Web Text))

Max Activating Token Index: 378

Text #16

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.1855. Min Act: -0.0000

Data Index: 730033 (C4 (Web Text))

Max Activating Token Index: 505

Text #17

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.1510. Min Act: -0.0000

Data Index: 173649 (C4 (Web Text))

Max Activating Token Index: 599

Text #18

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.1055. Min Act: -0.0000

Data Index: 815464 (C4 (Web Text))

Max Activating Token Index: 399

Text #19

Max Range: 1.2659. Min Range: -1.2659

Max Act: 1.0602. Min Act: -0.0000

Data Index: 943451 (C4 (Web Text))

Max Activating Token Index: 674