Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3631 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.7134. Min Range: -1.7134

Max Act: 1.7134. Min Act: -0.0000

Data Index: 690786 (C4 (Web Text))

Max Activating Token Index: 181

Text #1

Max Range: 1.7134. Min Range: -1.7134

Max Act: 1.5607. Min Act: -0.0000

Data Index: 507892 (C4 (Web Text))

Max Activating Token Index: 164

Text #2

Max Range: 1.7134. Min Range: -1.7134

Max Act: 1.2618. Min Act: 0.0000

Data Index: 1291522 (C4 (Web Text))

Max Activating Token Index: 44

Text #3

Max Range: 1.7134. Min Range: -1.7134

Max Act: 1.0781. Min Act: -0.0000

Data Index: 895773 (C4 (Web Text))

Max Activating Token Index: 699

Text #4

Max Range: 1.7134. Min Range: -1.7134

Max Act: 1.0063. Min Act: -0.0000

Data Index: 1059110 (C4 (Web Text))

Max Activating Token Index: 409

Text #5

Max Range: 1.7134. Min Range: -1.7134

Max Act: 0.9469. Min Act: -0.0000

Data Index: 910823 (C4 (Web Text))

Max Activating Token Index: 698

Text #6

Max Range: 1.7134. Min Range: -1.7134

Max Act: 1.0455. Min Act: -0.0000

Data Index: 171963 (C4 (Web Text))

Max Activating Token Index: 297

Text #7

Max Range: 1.7134. Min Range: -1.7134

Max Act: 0.9684. Min Act: -0.0000

Data Index: 213840 (C4 (Web Text))

Max Activating Token Index: 575

Text #8

Max Range: 1.7134. Min Range: -1.7134

Max Act: 0.9591. Min Act: -0.0000

Data Index: 281987 (C4 (Web Text))

Max Activating Token Index: 828

Text #9

Max Range: 1.7134. Min Range: -1.7134

Max Act: 0.8706. Min Act: -0.0000

Data Index: 342751 (C4 (Web Text))

Max Activating Token Index: 663

Text #10

Max Range: 1.7134. Min Range: -1.7134

Max Act: 0.9231. Min Act: -0.0000

Data Index: 745151 (C4 (Web Text))

Max Activating Token Index: 74

Text #11

Max Range: 1.7134. Min Range: -1.7134

Max Act: 0.9098. Min Act: -0.0000

Data Index: 164543 (C4 (Web Text))

Max Activating Token Index: 467

Text #12

Max Range: 1.7134. Min Range: -1.7134

Max Act: 0.9093. Min Act: -0.0000

Data Index: 1066420 (C4 (Web Text))

Max Activating Token Index: 916

Text #13

Max Range: 1.7134. Min Range: -1.7134

Max Act: 0.8201. Min Act: -0.0000

Data Index: 1330582 (C4 (Web Text))

Max Activating Token Index: 395

Text #14

Max Range: 1.7134. Min Range: -1.7134

Max Act: 0.8784. Min Act: -0.0000

Data Index: 435039 (C4 (Web Text))

Max Activating Token Index: 332

Text #15

Max Range: 1.7134. Min Range: -1.7134

Max Act: 0.8609. Min Act: -0.0000

Data Index: 508610 (C4 (Web Text))

Max Activating Token Index: 452

Text #16

Max Range: 1.7134. Min Range: -1.7134

Max Act: 0.7863. Min Act: -0.0000

Data Index: 349389 (C4 (Web Text))

Max Activating Token Index: 989

Text #17

Max Range: 1.7134. Min Range: -1.7134

Max Act: 0.7830. Min Act: -0.0000

Data Index: 1277073 (C4 (Web Text))

Max Activating Token Index: 783

Text #18

Max Range: 1.7134. Min Range: -1.7134

Max Act: 0.8263. Min Act: -0.0000

Data Index: 1231009 (C4 (Web Text))

Max Activating Token Index: 895

Text #19

Max Range: 1.7134. Min Range: -1.7134

Max Act: 0.8703. Min Act: -0.0000

Data Index: 291536 (C4 (Web Text))

Max Activating Token Index: 1017