Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 682 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.8976. Min Act: -0.0000

Data Index: 47072 (C4 (Web Text))

Max Activating Token Index: 908

Text #1

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.8018. Min Act: -0.0000

Data Index: 356647 (C4 (Web Text))

Max Activating Token Index: 985

Text #2

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.8215. Min Act: -0.0000

Data Index: 264249 (C4 (Web Text))

Max Activating Token Index: 550

Text #3

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.6273. Min Act: -0.0000

Data Index: 111779 (C4 (Web Text))

Max Activating Token Index: 88

Text #4

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.6676. Min Act: -0.0000

Data Index: 808073 (C4 (Web Text))

Max Activating Token Index: 713

Text #5

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.8294. Min Act: -0.0000

Data Index: 1259394 (C4 (Web Text))

Max Activating Token Index: 207

Text #6

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.6397. Min Act: -0.0000

Data Index: 514708 (C4 (Web Text))

Max Activating Token Index: 974

Text #7

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.7128. Min Act: -0.0000

Data Index: 411387 (C4 (Web Text))

Max Activating Token Index: 960

Text #8

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.4809. Min Act: -0.0000

Data Index: 1038922 (C4 (Web Text))

Max Activating Token Index: 717

Text #9

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.6415. Min Act: -0.0000

Data Index: 1116022 (C4 (Web Text))

Max Activating Token Index: 801

Text #10

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.5343. Min Act: -0.0000

Data Index: 1200161 (C4 (Web Text))

Max Activating Token Index: 472

Text #11

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.3953. Min Act: -0.0000

Data Index: 754568 (C4 (Web Text))

Max Activating Token Index: 248

Text #12

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.4149. Min Act: -0.0000

Data Index: 564968 (C4 (Web Text))

Max Activating Token Index: 747

Text #13

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.4215. Min Act: -0.0000

Data Index: 521375 (C4 (Web Text))

Max Activating Token Index: 971

Text #14

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.4827. Min Act: -0.0000

Data Index: 988022 (C4 (Web Text))

Max Activating Token Index: 60

Text #15

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.3704. Min Act: -0.0000

Data Index: 256742 (C4 (Web Text))

Max Activating Token Index: 868

Text #16

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.3701. Min Act: -0.0000

Data Index: 1027329 (C4 (Web Text))

Max Activating Token Index: 985

Text #17

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.3599. Min Act: -0.0000

Data Index: 457993 (C4 (Web Text))

Max Activating Token Index: 29

Text #18

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.3460. Min Act: -0.0000

Data Index: 1054292 (C4 (Web Text))

Max Activating Token Index: 961

Text #19

Max Range: 1.8976. Min Range: -1.8976

Max Act: 1.4252. Min Act: -0.0000

Data Index: 315915 (C4 (Web Text))

Max Activating Token Index: 331