Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5278 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.4968. Min Range: -1.4968

Max Act: 1.4968. Min Act: -0.0000

Data Index: 552823 (C4 (Web Text))

Max Activating Token Index: 989

Text #1

Max Range: 1.4968. Min Range: -1.4968

Max Act: 1.3137. Min Act: -0.0000

Data Index: 1332664 (C4 (Web Text))

Max Activating Token Index: 385

Text #2

Max Range: 1.4968. Min Range: -1.4968

Max Act: 1.2316. Min Act: -0.0000

Data Index: 1337400 (C4 (Web Text))

Max Activating Token Index: 766

Text #3

Max Range: 1.4968. Min Range: -1.4968

Max Act: 1.2675. Min Act: -0.0000

Data Index: 576545 (C4 (Web Text))

Max Activating Token Index: 315

Text #4

Max Range: 1.4968. Min Range: -1.4968

Max Act: 1.1869. Min Act: -0.0000

Data Index: 10446 (C4 (Web Text))

Max Activating Token Index: 413

Text #5

Max Range: 1.4968. Min Range: -1.4968

Max Act: 1.2435. Min Act: -0.0000

Data Index: 1080986 (C4 (Web Text))

Max Activating Token Index: 400

Text #6

Max Range: 1.4968. Min Range: -1.4968

Max Act: 0.9432. Min Act: -0.0000

Data Index: 463837 (C4 (Web Text))

Max Activating Token Index: 192

Text #7

Max Range: 1.4968. Min Range: -1.4968

Max Act: 0.9443. Min Act: -0.0000

Data Index: 671308 (C4 (Web Text))

Max Activating Token Index: 838

Text #8

Max Range: 1.4968. Min Range: -1.4968

Max Act: 0.8916. Min Act: -0.0000

Data Index: 1066151 (C4 (Web Text))

Max Activating Token Index: 696

Text #9

Max Range: 1.4968. Min Range: -1.4968

Max Act: 0.8342. Min Act: -0.0000

Data Index: 991891 (C4 (Web Text))

Max Activating Token Index: 1007

Text #10

Max Range: 1.4968. Min Range: -1.4968

Max Act: 0.8078. Min Act: -0.0000

Data Index: 63397 (C4 (Web Text))

Max Activating Token Index: 816

Text #11

Max Range: 1.4968. Min Range: -1.4968

Max Act: 0.8929. Min Act: -0.0000

Data Index: 1070402 (C4 (Web Text))

Max Activating Token Index: 916

Text #12

Max Range: 1.4968. Min Range: -1.4968

Max Act: 0.8588. Min Act: -0.0000

Data Index: 822901 (C4 (Web Text))

Max Activating Token Index: 760

Text #13

Max Range: 1.4968. Min Range: -1.4968

Max Act: 0.8294. Min Act: -0.0000

Data Index: 91584 (C4 (Web Text))

Max Activating Token Index: 729

Text #14

Max Range: 1.4968. Min Range: -1.4968

Max Act: 0.8440. Min Act: -0.0000

Data Index: 812565 (C4 (Web Text))

Max Activating Token Index: 604

Text #15

Max Range: 1.4968. Min Range: -1.4968

Max Act: 0.8391. Min Act: -0.0000

Data Index: 1060946 (C4 (Web Text))

Max Activating Token Index: 175

Text #16

Max Range: 1.4968. Min Range: -1.4968

Max Act: 0.8593. Min Act: -0.0000

Data Index: 1241072 (C4 (Web Text))

Max Activating Token Index: 813

Text #17

Max Range: 1.4968. Min Range: -1.4968

Max Act: 0.7763. Min Act: -0.0000

Data Index: 988513 (C4 (Web Text))

Max Activating Token Index: 682

Text #18

Max Range: 1.4968. Min Range: -1.4968

Max Act: 0.8667. Min Act: -0.0000

Data Index: 148645 (C4 (Web Text))

Max Activating Token Index: 185

Text #19

Max Range: 1.4968. Min Range: -1.4968

Max Act: 0.8312. Min Act: -0.0000

Data Index: 873189 (C4 (Web Text))

Max Activating Token Index: 147