Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3546 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.0723. Min Range: -1.0723

Max Act: 1.0723. Min Act: -0.0000

Data Index: 609658 (C4 (Web Text))

Max Activating Token Index: 896

Text #1

Max Range: 1.0723. Min Range: -1.0723

Max Act: 1.0244. Min Act: -0.0000

Data Index: 1283191 (C4 (Web Text))

Max Activating Token Index: 667

Text #2

Max Range: 1.0723. Min Range: -1.0723

Max Act: 1.0365. Min Act: -0.0000

Data Index: 246290 (C4 (Web Text))

Max Activating Token Index: 569

Text #3

Max Range: 1.0723. Min Range: -1.0723

Max Act: 0.9802. Min Act: -0.0000

Data Index: 1346309 (C4 (Web Text))

Max Activating Token Index: 723

Text #4

Max Range: 1.0723. Min Range: -1.0723

Max Act: 1.0664. Min Act: -0.0000

Data Index: 516150 (C4 (Web Text))

Max Activating Token Index: 74

Text #5

Max Range: 1.0723. Min Range: -1.0723

Max Act: 0.9535. Min Act: -0.0000

Data Index: 57433 (C4 (Web Text))

Max Activating Token Index: 809

Text #6

Max Range: 1.0723. Min Range: -1.0723

Max Act: 0.9050. Min Act: -0.0000

Data Index: 165310 (C4 (Web Text))

Max Activating Token Index: 89

Text #7

Max Range: 1.0723. Min Range: -1.0723

Max Act: 0.7995. Min Act: -0.0000

Data Index: 1219501 (C4 (Web Text))

Max Activating Token Index: 286

Text #8

Max Range: 1.0723. Min Range: -1.0723

Max Act: 0.8557. Min Act: -0.0000

Data Index: 881814 (C4 (Web Text))

Max Activating Token Index: 1023

Text #9

Max Range: 1.0723. Min Range: -1.0723

Max Act: 0.9275. Min Act: -0.0000

Data Index: 18386 (C4 (Web Text))

Max Activating Token Index: 188

Text #10

Max Range: 1.0723. Min Range: -1.0723

Max Act: 0.7670. Min Act: -0.0000

Data Index: 601013 (C4 (Web Text))

Max Activating Token Index: 846

Text #11

Max Range: 1.0723. Min Range: -1.0723

Max Act: 0.7300. Min Act: -0.0000

Data Index: 932451 (C4 (Web Text))

Max Activating Token Index: 523

Text #12

Max Range: 1.0723. Min Range: -1.0723

Max Act: 0.7712. Min Act: -0.0000

Data Index: 1009220 (C4 (Web Text))

Max Activating Token Index: 97

Text #13

Max Range: 1.0723. Min Range: -1.0723

Max Act: 0.7221. Min Act: -0.0000

Data Index: 330470 (C4 (Web Text))

Max Activating Token Index: 582

Text #14

Max Range: 1.0723. Min Range: -1.0723

Max Act: 0.7684. Min Act: -0.0000

Data Index: 214914 (C4 (Web Text))

Max Activating Token Index: 365

Text #15

Max Range: 1.0723. Min Range: -1.0723

Max Act: 0.7503. Min Act: -0.0000

Data Index: 6446 (C4 (Web Text))

Max Activating Token Index: 581

Text #16

Max Range: 1.0723. Min Range: -1.0723

Max Act: 0.6785. Min Act: -0.0000

Data Index: 1066151 (C4 (Web Text))

Max Activating Token Index: 634

Text #17

Max Range: 1.0723. Min Range: -1.0723

Max Act: 0.7158. Min Act: -0.0000

Data Index: 87632 (C4 (Web Text))

Max Activating Token Index: 470

Text #18

Max Range: 1.0723. Min Range: -1.0723

Max Act: 0.7109. Min Act: -0.0000

Data Index: 1327135 (C4 (Web Text))

Max Activating Token Index: 391

Text #19

Max Range: 1.0723. Min Range: -1.0723

Max Act: 0.7142. Min Act: -0.0000

Data Index: 1209723 (C4 (Web Text))

Max Activating Token Index: 666