Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3746 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.7574. Min Act: -0.0001

Data Index: 1051035 (C4 (Web Text))

Max Activating Token Index: 215

Text #1

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.7237. Min Act: -0.0001

Data Index: 1105084 (C4 (Web Text))

Max Activating Token Index: 810

Text #2

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.7429. Min Act: -0.0001

Data Index: 1171402 (C4 (Web Text))

Max Activating Token Index: 417

Text #3

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.7040. Min Act: -0.0001

Data Index: 755542 (C4 (Web Text))

Max Activating Token Index: 437

Text #4

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.6922. Min Act: -0.0001

Data Index: 1298360 (C4 (Web Text))

Max Activating Token Index: 468

Text #5

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.7419. Min Act: -0.0001

Data Index: 755185 (C4 (Web Text))

Max Activating Token Index: 527

Text #6

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.7253. Min Act: -0.0001

Data Index: 32813 (C4 (Web Text))

Max Activating Token Index: 869

Text #7

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.6336. Min Act: -0.0001

Data Index: 316970 (C4 (Web Text))

Max Activating Token Index: 390

Text #8

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.7832. Min Act: -0.0001

Data Index: 167593 (C4 (Web Text))

Max Activating Token Index: 37

Text #9

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.6241. Min Act: -0.0001

Data Index: 1332607 (C4 (Web Text))

Max Activating Token Index: 834

Text #10

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.5675. Min Act: -0.0001

Data Index: 291720 (C4 (Web Text))

Max Activating Token Index: 685

Text #11

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.5734. Min Act: -0.0001

Data Index: 50985 (C4 (Web Text))

Max Activating Token Index: 114

Text #12

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.5919. Min Act: -0.0001

Data Index: 420021 (C4 (Web Text))

Max Activating Token Index: 726

Text #13

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.6759. Min Act: -0.0001

Data Index: 1259227 (C4 (Web Text))

Max Activating Token Index: 633

Text #14

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.6160. Min Act: -0.0001

Data Index: 107831 (C4 (Web Text))

Max Activating Token Index: 788

Text #15

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.7004. Min Act: -0.0001

Data Index: 883321 (C4 (Web Text))

Max Activating Token Index: 574

Text #16

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.6164. Min Act: -0.0001

Data Index: 1282947 (C4 (Web Text))

Max Activating Token Index: 546

Text #17

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.6908. Min Act: -0.0001

Data Index: 440727 (C4 (Web Text))

Max Activating Token Index: 518

Text #18

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.7049. Min Act: -0.0001

Data Index: 946898 (C4 (Web Text))

Max Activating Token Index: 999

Text #19

Max Range: 0.7832. Min Range: -0.7832

Max Act: 0.6082. Min Act: -0.0001

Data Index: 471501 (C4 (Web Text))

Max Activating Token Index: 459