Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 686 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.0740. Min Range: -1.0740

Max Act: 1.0740. Min Act: -0.0000

Data Index: 865419 (C4 (Web Text))

Max Activating Token Index: 958

Text #1

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.9665. Min Act: -0.0000

Data Index: 1185503 (C4 (Web Text))

Max Activating Token Index: 657

Text #2

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.9182. Min Act: -0.0000

Data Index: 180828 (C4 (Web Text))

Max Activating Token Index: 806

Text #3

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.9361. Min Act: -0.0000

Data Index: 1155975 (C4 (Web Text))

Max Activating Token Index: 905

Text #4

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.8791. Min Act: -0.0000

Data Index: 195029 (C4 (Web Text))

Max Activating Token Index: 785

Text #5

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.8784. Min Act: -0.0000

Data Index: 1157179 (C4 (Web Text))

Max Activating Token Index: 556

Text #6

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.8728. Min Act: -0.0000

Data Index: 169642 (C4 (Web Text))

Max Activating Token Index: 966

Text #7

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.9075. Min Act: -0.0000

Data Index: 222444 (C4 (Web Text))

Max Activating Token Index: 498

Text #8

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.8390. Min Act: -0.0000

Data Index: 396996 (C4 (Web Text))

Max Activating Token Index: 845

Text #9

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.8343. Min Act: -0.0000

Data Index: 845893 (C4 (Web Text))

Max Activating Token Index: 994

Text #10

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.8681. Min Act: -0.0000

Data Index: 782178 (C4 (Web Text))

Max Activating Token Index: 911

Text #11

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.8532. Min Act: -0.0000

Data Index: 784218 (C4 (Web Text))

Max Activating Token Index: 811

Text #12

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.8319. Min Act: -0.0000

Data Index: 816896 (C4 (Web Text))

Max Activating Token Index: 622

Text #13

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.8654. Min Act: -0.0000

Data Index: 463861 (C4 (Web Text))

Max Activating Token Index: 516

Text #14

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.8555. Min Act: -0.0000

Data Index: 1147227 (C4 (Web Text))

Max Activating Token Index: 617

Text #15

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.8024. Min Act: -0.0000

Data Index: 794763 (C4 (Web Text))

Max Activating Token Index: 786

Text #16

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.8659. Min Act: -0.0000

Data Index: 1240016 (C4 (Web Text))

Max Activating Token Index: 865

Text #17

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.8672. Min Act: -0.0000

Data Index: 1129372 (C4 (Web Text))

Max Activating Token Index: 448

Text #18

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.8379. Min Act: -0.0000

Data Index: 429089 (C4 (Web Text))

Max Activating Token Index: 615

Text #19

Max Range: 1.0740. Min Range: -1.0740

Max Act: 0.8991. Min Act: -0.0000

Data Index: 214364 (C4 (Web Text))

Max Activating Token Index: 641