Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4379 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.0186. Min Range: -1.0186

Max Act: 1.0186. Min Act: -0.0000

Data Index: 431162 (C4 (Web Text))

Max Activating Token Index: 509

Text #1

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.8316. Min Act: -0.0000

Data Index: 751818 (C4 (Web Text))

Max Activating Token Index: 848

Text #2

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.8197. Min Act: -0.0000

Data Index: 721988 (C4 (Web Text))

Max Activating Token Index: 897

Text #3

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.8442. Min Act: -0.0000

Data Index: 712567 (C4 (Web Text))

Max Activating Token Index: 615

Text #4

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.7812. Min Act: -0.0000

Data Index: 104023 (C4 (Web Text))

Max Activating Token Index: 520

Text #5

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.6791. Min Act: -0.0000

Data Index: 611254 (C4 (Web Text))

Max Activating Token Index: 666

Text #6

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.7073. Min Act: -0.0000

Data Index: 657854 (C4 (Web Text))

Max Activating Token Index: 390

Text #7

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.7112. Min Act: -0.0000

Data Index: 434431 (C4 (Web Text))

Max Activating Token Index: 375

Text #8

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.6538. Min Act: -0.0000

Data Index: 35751 (C4 (Web Text))

Max Activating Token Index: 456

Text #9

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.6747. Min Act: -0.0000

Data Index: 1036408 (C4 (Web Text))

Max Activating Token Index: 273

Text #10

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.6569. Min Act: -0.0000

Data Index: 869589 (C4 (Web Text))

Max Activating Token Index: 795

Text #11

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.6463. Min Act: -0.0000

Data Index: 543968 (C4 (Web Text))

Max Activating Token Index: 471

Text #12

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.7101. Min Act: -0.0000

Data Index: 534727 (C4 (Web Text))

Max Activating Token Index: 251

Text #13

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.5745. Min Act: -0.0000

Data Index: 551435 (C4 (Web Text))

Max Activating Token Index: 96

Text #14

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.6012. Min Act: -0.0000

Data Index: 659263 (C4 (Web Text))

Max Activating Token Index: 617

Text #15

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.6176. Min Act: -0.0000

Data Index: 357398 (C4 (Web Text))

Max Activating Token Index: 640

Text #16

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.6149. Min Act: -0.0000

Data Index: 384724 (C4 (Web Text))

Max Activating Token Index: 633

Text #17

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.5799. Min Act: -0.0000

Data Index: 1011255 (C4 (Web Text))

Max Activating Token Index: 130

Text #18

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.5954. Min Act: -0.0000

Data Index: 883878 (C4 (Web Text))

Max Activating Token Index: 766

Text #19

Max Range: 1.0186. Min Range: -1.0186

Max Act: 0.5374. Min Act: -0.0000

Data Index: 1309592 (C4 (Web Text))

Max Activating Token Index: 396