Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2575 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.7700. Min Act: -0.0000

Data Index: 1115077 (C4 (Web Text))

Max Activating Token Index: 582

Text #1

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.3827. Min Act: -0.0000

Data Index: 656241 (C4 (Web Text))

Max Activating Token Index: 73

Text #2

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.3224. Min Act: -0.0000

Data Index: 142983 (C4 (Web Text))

Max Activating Token Index: 617

Text #3

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.0940. Min Act: -0.0000

Data Index: 474905 (C4 (Web Text))

Max Activating Token Index: 307

Text #4

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.3036. Min Act: -0.0000

Data Index: 721177 (C4 (Web Text))

Max Activating Token Index: 950

Text #5

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.2865. Min Act: -0.0000

Data Index: 254052 (C4 (Web Text))

Max Activating Token Index: 575

Text #6

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.3784. Min Act: -0.0000

Data Index: 980749 (C4 (Web Text))

Max Activating Token Index: 750

Text #7

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.2627. Min Act: -0.0000

Data Index: 677310 (C4 (Web Text))

Max Activating Token Index: 144

Text #8

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.2545. Min Act: -0.0000

Data Index: 441685 (C4 (Web Text))

Max Activating Token Index: 491

Text #9

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.3476. Min Act: -0.0000

Data Index: 1188347 (C4 (Web Text))

Max Activating Token Index: 900

Text #10

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.1749. Min Act: -0.0000

Data Index: 819504 (C4 (Web Text))

Max Activating Token Index: 393

Text #11

Max Range: 2.7700. Min Range: -2.7700

Max Act: 1.9595. Min Act: -0.0000

Data Index: 1304251 (C4 (Web Text))

Max Activating Token Index: 628

Text #12

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.1767. Min Act: -0.0000

Data Index: 593802 (C4 (Web Text))

Max Activating Token Index: 976

Text #13

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.2518. Min Act: -0.0000

Data Index: 968922 (C4 (Web Text))

Max Activating Token Index: 750

Text #14

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.1063. Min Act: -0.0000

Data Index: 1093233 (C4 (Web Text))

Max Activating Token Index: 282

Text #15

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.1259. Min Act: -0.0000

Data Index: 730633 (C4 (Web Text))

Max Activating Token Index: 994

Text #16

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.3281. Min Act: -0.0000

Data Index: 419527 (C4 (Web Text))

Max Activating Token Index: 750

Text #17

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.2074. Min Act: -0.0000

Data Index: 22059 (C4 (Web Text))

Max Activating Token Index: 930

Text #18

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.0808. Min Act: -0.0000

Data Index: 1305645 (C4 (Web Text))

Max Activating Token Index: 12

Text #19

Max Range: 2.7700. Min Range: -2.7700

Max Act: 2.1807. Min Act: -0.0000

Data Index: 1269255 (C4 (Web Text))

Max Activating Token Index: 576